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•  Interrogation  of  the  3-M  data  base  for  an  F-14  squadron 

•  Selection  of  "strong”  and  "weak"  subsystems  to  be  investigated, 
based  on  maintenance  error  rates  and  elapsed  maintenance  time 
(EMT) 

•  Interviews  of  subject  matter  experts  to  ascertain  the  influence 
of  design  features  Cor  the  lack  thereof)  on  the  subsystems  thus 
identified 

•  Collection,  reduction,  and  analysis  of  data 

An  additional  project  task  was  to  develop  a  method  to  evaluate  the  im¬ 
pact  of  improvements  in  maintenance.  This  method  utilized  the  Aircraft 
Maintenance  Effectiveness  Simulation  CAHES)  Model. 
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PREFACE 


Maintenance  in  today's  era  of  high  technology  is  a  major 
problem.  It  has  been  estimated  to  comprise  in  excess  of  50 
percent  of  an  operational  weapons  system's  operating  costs, 
with  signs  that  it  is  continuing  to  increase  its  share.  As 
a  result  of  this,  the  Navy  is  now  giving  increased  attention 
to  a  wide  range  of  approaches  in  which  the  quality  of  weapon 
system  maintenance  can  be  improved. 

A  primary  issue  in  that  connection  is  the  matter  of  design 
for  the  maintainer.  To  date,  the  design- for-maintainer 
issue  has  been  largely  ignored  by  the  behavioral  sciences 
community  in  the  Navy.  The  little  attention  given  to 
design- for-maintainer  research  has  been  both  sporadic  and 
shallow.  One  reason  for  this  has  been  the  lack  of  a  mecha¬ 
nism  for  closing  the  feedback  loop  between  the  operational 
environment  and  the  R  &  D  community,  and  for  quantitatively 
establishing  the  impact  of  design- for-maintainer  on  weapon 
system  operational  readiness  and  operating  costs.  Develop¬ 
ment  of  such  a  mechanism  was  the  goal  of  this  project. 

This  report  documents  all  work  performed  by  Xyzyx  Infor¬ 
mation  Corporation  for  the  Naval  Air  Development  Center, 
under  contract  #N62269-80-C-0215.  The  report  is  organized 
as  follows: 

•  Section  I  provides  an  overview  of  the  problem,  describ¬ 
ing  various  factors  in  the  maintenance  environment  that 
affect  system  performance. 

•  Section  II  describes  quantitative  aspects  of  the 
problem  and  various  research  approaches. 

•  Section  III  describes  the  method  developed  to  identify 
design  features  influencing  maintenance. 

•  Section  IV  describes  the  method  used  in  evaluating  the 
impact  of  improvements  in  maintenance. 

J 

•  Section  V  documents  the  results  obtained  during  the 
course  of  the  project. 

•  Section  VI  presents  the  conclusions,  implications,  and 
recommendations  drawn  from  these  results. 
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SECTION  I 

OVERVIEW  OF  THE  PROBLEM 


The  design- for-maintainer  problem  is  described  under  the 
following  major  headings: 

•  Introduction 

•  Maintenance  Errors 

•  System  Design  and  Support  Factors  Affecting  Maintenance 


INTRODUCTION 

The  goal  of  Navy  Maintenance  is  to  keep  weapon  systems 
operationally  ready  in  an  efficient  manner.  Unfortunately, 
there  are  indications  that  this  goal,  while  being  approach¬ 
ed,  is  not  being  met.  That  is,  a  number  of  weapon  systems 
are  not  being  kept  in  an  operationally  ready  state  for  an 
adequate  percentage  of  the  time. 

In  the  paragraphs  that  follow,  this  problem  is  discussed 
further  in  terms  of  1)  operational  readiness,  2)  cost  of 
maintenance,  and  3)  measures  of  maintenance  technician 
effectiveness . 


Operational  Readiness 


Operational  readiness  (OR)  data  are  presented  for  two  cur¬ 
rent  aircraft  models,  the  F-14A  and  the  SH-2F.  As  shown  in 
Table  1,  the  F-14A  has  an  operational  readiness  of  only 
about  58  percent.  However,  even  this  figure  is  somewhat 
misleading.  When  only  the  full  systems  capability  (FSC)  is 
considered,  operational  readiness  reduces  to  about  52  per¬ 
cent. 


There  is  some  indication  that  the  52  to  58  percent  figure 
for  operational  readiness  is  not  unusually  low.  Experts 
indicate  that  other  weapon  systems  are  experiencing  the  same 
or  even  lower  operational  readiness.  Consider  the  SH-2F. 
Operational  readiness  data  for  that  aircraft  are  presented 
in  Table  2. 

At  first  glance,  the  SH-2F  appears  to  have  a  much  higher 
level  of  operational  readiness,  (71  percent,  as  compared  to 
58  percent  for  the  F-14A) .  However,  when  only  the  FSC 
portion  of  operational  readiness  is  considered,  SH-2F  OR 
drops  to  49  percent,  which  is  about  the  same  as  the  F-14A. 
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TABLE  1.  ANALYSIS  OF  F-14A  OPERATIONAL  READINESS 


RMCM-S 

(Reduced  Material  Condition  for 
Maintenance  -  Scheduled) 

0% 

RMCM-U 

(Reduced  Material  Condition  for 
Maintenance  -  Unscheduled) 

3% 

NFE 

{Not  Fully  Equipped) 

3% 

FSC 

(Full  Systems  Capability) 

52% 

Total -Operational  Readiness  (OR) 

58% 

TABLE  2.  ANALYSIS  OF  SH-2F  OPERATIONAL  READINESS 


RMCM - S 

0% 

i 

RMCM  -U 

3% 

j 

NFE 

19% 

1 

FSC 

49% 

J 

« 

| 

OR 

71% 

J 
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The  difference  is  apparently  due  to  the  fact  that  the  SH-2F 
is  basically  a  single  aircraft  detachment.  Therefore,  it 
often  flies  a  mission  without  being  fully  equipped  — 
wherein  all  the  functional  requirements  probably  cannot 
and/or  are  not  being  met. 

Only  a  few  decades  ago,  missle  systems  were  expected  and 
able  to  achieve  an  OR  figure  of  80  percent  or  higher. 

Often,  the  OR  percentage  was  in  the  90s.  An  operational 
readiness  of  49  to  58  percent  in  the  systems  of  today 
represents  a  decline  of  significant  proportions. 


Cost  of  Maintenance 


The  true  cost  of  maintenance  inadequacies  is  reflected  in 
the  Not  Operationally  Ready  (NOR)  parameter.  NOR  means  that 
the  aircraft  has  lost  its  value  as  a  weapon  system  during  a 
particular  period  of  time.  Thus,  the  NOR  aircraft  represent 
an  important  cost  item. 

Each  F-14A  is  priced  at  $21  million.  However,  operational 
readiness  is  such  that  this  $21  million  delivers  only  0.58 
aircraft  per  year.  The  NOR  proportion  thus  equals  .42. 
Assuming  15  operational  squadrons  and  11  aircraft  per 
squadron,  the  value  of  the  NOR  aircraft  inventory  is  15 
squadrons  x  4.62  NOR  aircraft  x  $21  million  =  $1,455 
billion . 

A  similar  estimate  can  be  made  for  the  forthcoming  LAMPS. 
According  to  the  Los  Angeles  Times,  the  Navy  plans  to  ac¬ 
quire  204  LAMPS  Mark  III  Helicopters  for  $3.5  billion. 

Using  the  SH-2F  OR/NOR  data,  the  value  of  the  NOR  inventory 
will  be  $3.5  x  0.29  =  $1,015  billion.  This  NOR  value  in¬ 
creases  considerably  if  the  NFE  is  included  as  partial  NOR. 
If  we  consider  half  the  NFE  in  our  analysis,  the  NOR  in¬ 
creases  to  38.5  percent  and  the  value  of  the  NOR  inventory 
increases  to  $1.35  billion. 

Obviously,  the  value  of  NOR  aircraft  is  quite  large.  Even  a 
small  reduction  of  the  NOR  inventory  represents  considerable 
savings.  An  important  question  is  whether  there  is  any  in¬ 
dication  or  evidence  that  better  design  for  the  maintainer 
can  reduce  the  NOR  inventory. 
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Measures  of  Maintenance  Technician  Effectiveness 

Given  that  maintenance  costs  are  increasing  while  weapon 
system  availability  is  declining,  a  question  arises  as  to 
the  possible  causes.  One  thesis  of  this  project  is  that  one 
of  the  major  contributors  to  the  problem  is  the  maintenance 
technician.  He  is  performing  well  below  his  natural  poten¬ 
tial.  As  a  further  thesis,  it  is  contended  that  the  tech¬ 
nician's  performance  is  inhibited  by  certain  factors  in 
weapon  system • design. 

In  the  paragraphs  that  follow,  measures  of  effectiveness  are 
described  providing  support  for  these  contentions.  These 
measures  are  based  on  intensive  analysis  of  3-M  data  and  ex¬ 
perience  with  a  simulation  model  designed  to  process  the 
data.  The  measures  of  interest  are:  error  rate,  elapsed 
maintenance  time,  and  maintenance  manhours. 


Maintenance  Error  Rate 

Three  types  of  maintenance  errors  are  disclosed  by  3-M  data. 
These  are  Type  I  errors.  Type  II  errors,  and  Type  d  errors. 

A  Type  I  error  occurs  when  a  good  unit  is  removed  errone¬ 
ously.  A  Type  II  error  occurs  when  a  bad  or  malfunctioning 
unit  is  not  found,  which  often  results  in  the  wrong  thing 
being  repaired.  A  Type  d  error  occurs  when  the  maintenance 
activity  produces  damage  or  malfunction  in  the  system. 

The  different  types  of  maintenance  errors  indicate  different 
problems  associated  with  design  for  the  maintainer.  Type  I 
error  is  mostly  a  troubleshooting  problem.  That  is,  the 
technician  removes  a  good  item  because  he  cannot  locate  the 
cause  of  the  problem.  Associated  design  problems  relate 
primarily  to  the  inadequacies  of  information  relevant  to 
troubleshooting . 

Type  II  error  is  usually  committed  in  checkout.  That  is, 
the  technician  can't  verify  the  "squawk."  The  design  prob¬ 
lems  associated  with  checkout  are  similar  to  those  related 
to  troubleshooting 

Type  d  error  is  associated  primarily  with  handling.  It  is 
reflected  in  problems  of  accessibility,  exposed  components, 
inadequate  protection,  etc. 

These  different  types  of  errors  occur  with  varying  frequency 
depending  on  the  kinds  of  equipment  involved.  The  variation 
is  shown  in  Table  3,  where  error  rates  are  listed  for  the 
different  oraanizational  level  work  centers  supporting  the 
P-14A. 
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TABLE  3.  SUMMARY  OF  ERROR  RATES  FOR  THE  F-14A  WORK  CENTERS 


,  ERROR  RATE 

WORK  CENTER  TYPE  I  TYPE  II  &  d 


POWER  PLANTS  (110) 

5.9 

15.9 

AIRFRAME  (120) 

6.8 

16.0 

CORROSION 

CONTROL  (121) 

0. 

6.4 

AVIATOR 

EQUIP  (131) 

0. 

4.1 

SAFETY 

EQUIP  (132) 

17.8 

11.0 

ELECTRONICS  (210) 

13.0 

8.5 

ELECTRICAL 

INST  (220) 

17.7 

6.7 

ARMAMENTS  (230) 

5.4 

6.6 

ELECTRO-WEAPS 

CONTROL  (232) 

15.8 

7.3 

TROUBLESHOOTERS 

(320) 

0. 

12.5 

AVERAGE  FOR  ALL 

WORK  CENTERS 

14.5 

9.8 
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Note  that  Type  I  errors  are  more  prevalent  in  the  electronic 
and  electrical  work  centers,  such  as  132,  210,  220,  and  232. 
Conversely,  Type  II  and  Type  d  errors  predominate  in  those 
work  centers  concerned  with  mechanical  equipment  such  as 
power  plants  (110)  and  airframe  (120)  . 

The  analysis  by  subsystem  in  Table  4  is  even  more  revealing. 
There,  the  subsystem  with  the  highest  Type  I  error  rate  is 
seen  to  be  the  AWG-9.  The  AWG-9  has  been  extensively  auto¬ 
mated  to  support  maintenance.  The  data  indicate  that  much 
of  that  effort  may  have  been  counterproductive. 

The  AWG-9  On  Board  Computer/Buil t-In  Test  Equipment 
(OBC/BITE)  errs  12  percent  of  the  time.  In  addition,  the 
technicians  commit  Type  I  errors  21  percent  and  Type  II 
and/or  d  errors  7  percent  of  the  time.  This  could  mean  that 
technicians  learn  to  suspect  the  OBC/BITE  and  go  on  their 
own.  However,  what  the  technicians  do  not  realize  is  that 
their  own  error  rate  is  even  higher. 

Continuing  with  Table  4,  note  the  34  percent  Type  Il/d  rate 
for  the  TF-30  Engine.  Almost  all  of  this  is  Type  d  error. 
This  means  that  every  three  maintenance  actions  result  in 
one  or  more  malfunctions  (damage)  created  by  the  techni¬ 
cians.  Undoubtedly,  much  of  this  damage  is  due  to  hardware 
design  characteristics. 

Similar  problems  are  seen  in  the  SH-2F  data,  presented  in 
Table  5.  Although  the  damage  error  rate  for  the  T-58  Engine 
is  not  as  high  as  the  error  rate  for  the  TF-30  Engine,  it  is 
still  quite  high.  The  Type  d  error  rate  is  also  high  for 
the  Main  Rotor  System,  another  mechanical  system.  The  data 
tend  to  indicate  that  more  effective  design  for  the  main- 
tainer  could  have  a  significant  impact  on  the  OR/NOR  ratio. 

An  interesting  fact  about  Table  5  is  the  absence  of  any 
electronic  subsystems.  Although  the  concern  for  electronic 
maintenance  problems  in  general  is  valid,  the  data  indicate 
that  the  problems  in  other  areas  are  even  greater. 
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TA8LE  4.  ERROR  RATES  OF  THE  FIVE  F-14A  SUBSYSTEMS  WITH  THE  GREATEST  MAINTENANCE  BURDEN 


TABLE  5.  ERROR  RATES  OF  THE  FIVE  SH-2F  SUBSYSTEMS  WITH  THE  GREATEST  MAINTENANCE  BURDEN 
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that  after  a  typical  sortie,  the  bird  returns  witn  an 
average  requirement  for  4.37  corrective  maintenance 
actions. 


NADC-79218-60 


Elapsed  Maintenance  Time  (EMT) 

The  term  EMT  refers  to  the  time  the  aircraft  is  down  for 
maintenance  of  a  particular  system.  It  denotes  active  main¬ 
tenance  time  only  and  does  not  include  delays  due  to  lack  of 
other  support.  When  examining  EMT,  it  is  useful  to  present 
the  data  in  the  form  of  a  curve  representing  cumulative 
actions  versus  elapsed  time.  Such  a  curve  is  shown  in 
Figure  1. 

This  curve  was  derived  from  the  3-M  data  base  relative  to 
the  F-14A  Flight  Hydraulic  Power  System.  Two  points  may  be 
made  from  inspection  of  the  data  shown  here.  One  concerns 
elapsed  time  as  such.  The  other  concerns  the  shape  of  the 
EMT  curve. 

Elapsed  Time.  Half  the  maintenance  actions  are  completed  in 
an  EMT  of  2.25  hours  or  less.  That  value  appears  reason¬ 
able.  But  at  the  opposite  extreme,  a  quarter  of  the  actions 
require  between  4.2  and  12.0  hours  of  elapsed  time  on  the 
part  of  two  to  four  men.  That  value  seems  excessive. 
Experience  indicates  that  most  of  the  high  EMT  is  due  to 
errors  and  equipment  design.  Often,  the  two  are  related, 
i.e.,  poor  design  fosters  errors.  Unlike  the  Types  I,  II, 
and  d  errors,  these  errors  are  self-detected  and  corrected 
by  the  same  technician.  Thus,  they  do  not  get  counted  as 
errors.  Instead,  the  EMT  becomes  inflated  by  their 
presence . 

It  is  tempting  to  assume  that  the  limited  space  available  in 
the  F-14A  caused  packaging  and  accessibility  problems  which 
resulted  in  the  high  EMT  values.  Undoubtedly,  dense 
packaging  resulting  from  space  limitation  is  a  major 
contributor.  However,  the  equally  high  Type  d  errors  on  the 
SH-2F  would  indicate  that  the  problem  is  caused  by  far  more 
than  merely  dense  packaging. 


Shape  of  EMT  Curve.  For  many  years,  it  has  been  taken  for 
granted  that  EMT  is  distributed  exponentially.  Yet,  de- 
tailed  analyses  of  maintenance  tasks  or  steps  seldom  justify 
such  a  distribution.  That  is,  detailed  time-line  analyses 
of  the  tasks  required  of  technicians  seldom  provide  any 
clues  as  to  why  the  distribution  should  be  other  than 
normal.  The  time-line  data  usually  indicate  that  the  EMT 
I  should  be  considerably  shorter  than  what  we  are  currently 

experiencing. 


21 


203C.81.6 


NADC-792 18-60 


Maintenance  Manhours  (MMH) 

The  MMH  portion  of  the  3-M  data  bank  must  be  treated  very 
carefully.  Technicians  and/or  managers  tend  to  make  MMH 
entries  to  justify  overtime,  more  manpower,  etc.  Also,  the 
"buddy  system"  tends  to  inflate  the  MMH  data.  Nevertheless, 
relative  MMH  data  are  still  useful.  Consider  the  data  shown 
in  Table  6.  MMH  per  flight  hour  (FH)  are  compared  for  the 
F-14A  and  the  SH-2F. 

The  corrective  MMH  for  the  F-14A  is  over  2.6  times  greater 
than  the  corrective  MMH  for  the  SH-2F.  This  would  tend  to 
support  the  argument  that  a  major  contributor  to  the  problem 
is  the  space  limitations  of  the  F-14A. 

However,  there  probably  are  other  design  problems  common  to 
both  the  F-14A  and  the  SH-2F.  For  example,  even  with  the 
lower  MMH/FH,  the  Full  Systems  Capability  (FSC)  for  the 
SH-2F  (49%)  is  no  better  than  the  FSC  for  the  F-14A  (52%)  . 
This  would  indicate  that  the  SH-2F  requires  less  manpower 
but  the  effectiveness  of  maintenance  is  about  the  same. 

Probably  the  most  alarming  part  of  the  MMH/FH  ratio  is  how 
high  it  is.  For  every  flight  hour  of  the  F-14A,  an  equiva¬ 
lent  of  five  men  working  a  full  day  is  needed  to  maintain 
the  aircraft.  Even  for  the  SH-2F,  each  flight  hour  requires 
an  equivalent  of  three  men  working  a  full  day  to  maintain 
the  aircraft. 

Personnel  assignment  policies  alone  do  not  explain  the  ex¬ 
cessively  high  MMH/FH.  Design  characteristics  clearly 
should  be  investigated  as  contributors  to  the  problem. 


Summary 

The  cost  of  weapon  system  maintenance  is  rising  while  oper¬ 
ational  readiness  is  falling.  Evidence  has  been  presented 
suggesting  that  one  of  the  contributors  to  this  problem  is 
less- than-adequate  performance  by  the  maintenance  tech¬ 
nician.  It  is  contended  that  a  major  cause  of  inadequate 
technician  performance  is  the  design  of  the  weapon  system. 


MAINTENANCE  ERRORS 

In  the  past,  analysts  have  used  time-related  measurements 
(such  as  maintenance  manhours)  as  the  primary  measures  of 
maintenance  performance.  However,  as  previously  suggested, 
measurements  of  maintenance  errors  appear  to  provide  much 
greater  guidance  to  the  source  of  ineffective  maintenance 
practice . 
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PERCENT 

ACTIONS 


Figure  !•  EMT  Curve  for  the  Flight  Hydraulic  Power  System 


TABLE  6.  MMH/FH  COMPARISONS  FOR  THE  F-14A  AND  SH-2F 


F-14A 

SH-2F. 

TOTAL*  MMH/FH 

43.2 

24.7 

PLANNED  MAINT 

6.4 

3.2 

CORRECTIVE  MAINT 

16.4 

6.2 

Total  includes  all  support  actions  (SAF)  and  technical 
directive  compliance  (TDC)  as  well  as  planned  and 
corrective  maintenance. 
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Curiously,  little  systematic  study  has  been  applied  to  the 
problem  of  errors  committed  by  maintenance  technicians. 

Major  efforts  have  been  made  to  measure  and  control  per¬ 
formance  time,  but  performance  quality  has  gone  unques¬ 
tioned.  In  this  respect,  the  maintenance  research  community 
has  taken  the  same  path  as  was  taken  earlier  by  the  relia¬ 
bility  community.  Neither  has  constructed  a  viable  method 
of  dealing  with  the  errors  they  know  are  occurring. 

The  main  problem  has  been  in  finding  a  practical  way  of 
measuring  both  performance  time  and  performance  quality,  in¬ 
cluding  the  inter-relations  they  represent.  These  are  the 
elements  that  comprise  the  true  variable,  maintenance  effec¬ 
tiveness.  Measuring  performance  quality  has  rarely  been 
accompl ished .  Direct  detection  of  errors  by  special  observ¬ 
ers,  or  reliance  on  technicians  themselves  to  report  their 
own  errors,  has  not  been  practical. 

Xyzyx  has  devised  a  means  of  extracting  error  rates  from  3-M 
data.  Error  measurement  appears  entirely  feasible,  given 
the  real-world  data  of  the  quality  provided  by  the  3-M 
System.  All  of  the  error  measurements  described  here  are 
geared  to  records  normally  compiled  by  3-M,  and  identified 
by  number  in  the  3-M  Catalog. 

In  the  discussion  that  follows,  maintenance  errors  are 
examined  under  three  headings: 

•  Kinds  of  Errors 

•  Effects  of  Errors 

•  Measurement  of  Errors 

The  dynamics  of  errors  tend  to  be  similar  in  nature,  regard¬ 
less  of  the  nature  of  the  maintenance  environment.  However, 
the  discussion  here  applies  specifically  to  organizational 
maintenance  on  carrier-based  Navy  aircraft. 


Kinds  of  Errors 


Error  definition  was  introduced  earlier.  It  is  reiterated 
here  for  reader  convenience.  As  shown  in  Table  7,  mainte¬ 
nance  errors  may  be  conveniently  sorted  into  three  cate¬ 
gories:  Type  I,  Type  II,  and  Type  d. 

A  Type  I  error  occurs  when  the  technician  troubleshoots  the 
equipment  and  concludes  that  a  particular  unit  has  failed 
when,  in  fact,  it  has  not. 

A  Type  II  error  occurs  when  the  technician  checks  out  the 
equipment  and  concludes  that  it  is  okay  when,  in  fact,  it 
contains  a  defect. 
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TABLE  7.  KINDS  OF  MAINTENANCE  ERRORS 


TYPE  OF 

ERROR  EXPLANATION  OF  ERROR 


I  Technician  replaces  a  unit  that 

has  not  malfunctioned. 

II  Technician  fails  to  recognize  a 

unit  that  has  malfunctioned  or 
been  improperly  handled. 

d  Technician  fails  to  accomplish  a 

corrective  or  preventive  action 
properly. 
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TYPICAL 

SOURCE  OF  ERROR 


Troubleshooting 


Troubleshooting; 

checkout 


Removal  /Instal  lation  ; 
service;  repair;  adjust 
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A  Type  d  error  occurs  when  the  technician  damages  the  equip¬ 
ment  or  performs  a  corrective  or  preventive  maintenance  ac¬ 
tion  improperly.  Examples  would  be  loose  connectors,  mis¬ 
matched  parts,  improper  alignment,  and  inadequate  servicing. 

The  method  of  categorization  used  here  reflects  the  manner 
in  which  records  are  kept  in  the  3-M  System.  This  is  very 
important  to  the  overall  effort.  Any  measurement  method 
seeking  to  impact  the  Naval  air  maintenance  community  must 
be  as  consistent  as  possible  with  the  3-M  System  and  its 
associated  data  base. 

A  key  fact  seen  in  Table  7  is  that  each  type  of  error  tends 
to  be  associated  with  its  own  particular  maintenance  func¬ 
tion  or  functions.  This  link  will  be  referred  to  again 
later  in  the  report,  in  connection  with  both  design  factors 
and  data  processing. 


Effects  of  Errors 


The  effects  of  the  three  types  of  maintenance  errors  are 
summarized  in  Table  8.  As  shown  there,  errors  may  be  dis¬ 
covered  during  post- job  inspection,  before  a  flight,  or 
during  a  flight.  Following  discovery,  the  effects  may  be  to 
abort  the  flight,  repeat  the  maintenance  function,  use  spare 
parts  wastefully,  place  an  unnecessary  load  on  intermediate 
maintenance,  and,  of  course,  incur  a  risk  of  injury  or 
accident . 


Abort  Flight 

Other  than  the  risk  of  injury  or  accident,  which  is  present 
whenever  an  error  is  committed  in  aviation  maintenance,  the 
most  serious  effect  of  error  is  an  aborted  flight.  All 
equipment  items  do  not  share  equally  in  this  risk.  Flights 
are  aborted  only  when  critical  components  are  involved. 
Criticality  is  determined  by  pre-established  criteria.  Note 
that  errors  detected  and  corrected  during  post-job  inspec¬ 
tion  normally  do  not  cause  a  flight  to  be  aborted.  Note 
also  that  the  abort  effect  is  identical  for  all  three  types 
of  errors. 


Repeat  Function 

This  effect  points  up  the  cardinal  rule  of  maintenance  pro¬ 
ductivity:  "Every  error,  regardless  of  the  circumstances, 

creates  a  need  for  rework."  Errors  thus  represent  sheer 
dead  weight  in  the  system. 
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TABLE  8.  EFFECTS  OF  MAINTENANCE  ERRORS 


* 

■m 

Effects 

Kind 

Of 

When  Error 

Abort 

Repeat 

Use  Spare 

Place  Load 

Risk  Injury/ 

r- 

Is  Discovered 

Flight 

Function 

Wastefully 

On  IMA 

Or  Accident 

Post-Job 

•  (B) 
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• 
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Inspection 
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o 

If 

0  O 

Before 

Flight 

•  (A) 

•  (B) 
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a  cc 
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During 

Flight 
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Post-Job 
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Flight 
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Flight 
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Post-Job 
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•  (G) 

•  (D) 
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Inspection 
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Before 

•  (A) 

• 

•  (C) 

•  (D) 
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a  o 

IT  u 

Flight 

O  is 

cc  ,£ 

During 

•  (A) 

• 

•  (C) 

•  (0) 

• 

* 

*u 

Flight 

(A)  Flights  aborted  by  critical  equipment  items  only. 

(B)  "Function"'  includes  troubleshooting  &  repair. 

(C)  Spare  parts  wasted  only  when  broken. 

(D)  Load  placed  on  IMA  only  when  broken  part  repairable. 
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The  extent  of  the  necessary  rework  varies  with  the  type  of 
error.  For  Type  II  and  Type  d  errors,  rework  is  usually 
limited  to  the  offending  maintenance  function,  i.e.,  check¬ 
out  or  repair.  For  Type  I  errors,  however,  the  rework  is 
usually  more  extensive.  Not  only  does  troubleshooting  have 
to  be  repeated,  but  the  associated  corrective  action  must  be 
repeated . 

Rework  produces  two  kinds  of  negative  results.  First,  and 
most  obviously,  it  increases  the  direct  cost  of  maintenance 
by  requiring  the  presence  of  a  larger- than-optimum  staff. 
Second,  it  impedes  the  work  schedule,  thus  fostering  the 
creation  of  queues,  wherein  jobs  are  waiting  for  people.  In 
this  latter  sense,  excessive  demands  for  rework  always  im¬ 
pinge  upon  aircraft  availability. 


Ose  Spares  Wastefully 

Spare  parts  are  used  wastefully  in  two  of  the  three  types  of 
error  situations,  regardless  of  the  time  of  discovery.  The 
parts  problem  is  avoided  only  in  those  cases  where  the  error 
is  failing  to  see  that  a  replacement  is  necessary. 

The  greatest  waste  of  spare  parts  probabl-y  occurs  in  troub¬ 
leshooting  because  of  the  tr ial-and-error  approach  taken  by 
so  many  technicians.  It  is  true  that  good  units  removed  in 
error  may  be  used  again  after  being  recycled  through  Inter¬ 
mediate  Maintenance  Activity  (IMA).  Nevertheless,  while 
they  are  in  that  (recycling)  mode,  they  are  not  available 
for  re-issue.  Temporary  shortages  are  thus  generated,  often 
leading  to  reductions  in  aircraft  availability. 

The  waste  may  be  less  in  the  Type  d  (improper  repair)  cate¬ 
gory,  because  so  many  repairs  can  be  corrected  without  the 
need  for  new  parts.  However,  that  situation  is 
counterbalanced  by  the  occasional  repair  error  that  causes 
such  serious  damage  that  the  broken  parts  cannot  be  mended, 
even  in  IMA.  In  such  cases,  the  spares  inventory  is 
impacted  permanently. 


Place  Load  on  IMA 

Intermediate  maintenance  shops  are  burdened  unnecessarily 
every  time  a  part  is  replaced  in  error.  The  Type  I  error 
committed  in  troubleshooting  contributes  to  the  problem. 

IMA  must  handle  all  such  parts  as  though  they  were  defec¬ 
tive.  In  fact,  only  after  inspection  and/or  testing  at  IMA 
is  it  safe  for  the  parts  to  be  declared  ready  for  issue. 
Since  such  parts  enter  IMA  without  defect,  all  work  done  on 
them  must  be  regarded  as  wasted  effort.  The  result  is  an 
increase  in  the  direct  cost  of  maintenance. 
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The  dynamics  are  slightly  different  when  repair  is  done  in¬ 
correctly  at  the  organizational  level  (Type  Il/d  error)  .  In 
such  cases,  parts  go  to  IMA  only  if  they  are  damaged.  The 
nature  of  the  waste  lies  in  the  time  spent  by  IMA  in  mending 
the  broken  parts.  This  waste  is  relieved  only  when  the 
parts  are  broken  so  badly  that  they  cannot  be  repaired. 


Risk  of  Injury  or  Accident 

As  indicated  earlier,  the  risk  of  injury  or  accident  is 
present  every  time  an  error  is  committed  in  aircraft  main¬ 
tenance.  Depending  on  severity,  injuries  and  accidents  may 
impact  all  aspects  of  maintenance  and  operational  produc¬ 
tivity.  That  is,  they  can  affect  direct  costs,  aircraft 
availability,  personnel  availability,  the  spares  inventory, 
support  equipment  availability,  and  even  facility  availa¬ 
bility.  Injuries  and  accidents  are  thus  known  to  be  highly 
disruptive  of  normal  operations. 


Measurement  of  Errors 


All  of  the  error  measurements  described  here  can  be  obtained 
from  the  3-M  System  and  most  can  be  identified  by  number  in 
the  3-M  Catalog.  Where  necessary,  additional  procedures  are 
provided  by  Xyzyx  to  guide  the  analyst  in  extracting  error 
data  from  those  records. 


Type  I  Error  —  Good  Unit  Replaced 

Information  on  Type  I  errors  is  obtained  from  a  3-M  report 
generated  in  IMA.  The  report  is  MSOD  4790.A2551-01 ,  "No 
Defect  Item  Analysis  Summary."  This  report  summarizes 
equipment  items  processed  by  Aircraft  Intermediate  Main¬ 
tenance  Departments  for  which  reported  defects  could  not  be 
duplicated.  Such  items  thus  reflect  erroneous  removals  from 
the  aircraft. 


Type  II  Error  —  Bad  Unit  Not  Replaced 

Information  on  Type  II  errors  is  obtained  by  a  time-sequence 
analysis  of  3-M  records  CT11,  CT21  and  CT41.  The  analysis 
is  inferential  in  nature.  Where  successive  flights  are 
accompanied  by  corrective  maintenance  actions  on  the  same 
system,  and  those  actions  suddenly  stop,  a  particular  con¬ 
dition  may  be  inferred.  That  is,  the  final  corrective  ac¬ 
tion  was  successful,  but  each  preceding  action  must  have 
been  in  error.  Analysis  for  Type  II  errors  requires  engi¬ 
neering  knowledge  covering  system  equipment,  operating 
theory,  and  maintenance  practices. 
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Type  d  Error  —  Repair  Done  Incorrectly 

Information  on  Type  d  errors  is  obtained  from  malfunction 
codes  used  on  3-M  records  CT11  and  CT21.  Two  subsets  of  the 
Malfunction  Code  list  appear  in  Tables  9  and  10.  The  codes 
shown  in  Table  9  indicate  the  definite  occurrence  of  a  Type 
d  error.  The  codes  shown  in  Table  10  indicate  a  probable 
Type  d  error. 

Probable  errors  must  be  verified  through  reverse  time  se¬ 
quence  analysis  wherein  records  are  checked  for  prior  main¬ 
tenance  actions.  If  the  equipment  has  been  worked  on  in  the 
two  days  immediately  preceding  discovery  of  the  problem,  the 
defect  is  assumed  to  have  been  induced  by  maintenance  rather 
than  by  equipment  failure.  Engineering  knowledge  is  needed 
to  distinguish  between  equipment  failure  and  maintenance 
error . 


Error  Rate 

Error  rate  is  expressed  as  a  percentage  of  the  number  of 
relevant  corrective  maintenance  actions.  The  denominator 
includes  only  the  maintenance  actions  in  which  the  partic¬ 
ular  category  of  error  could  be  committed.  Thus,  the 
denominator  for  Types  I  and  II  errors  are  all  maintenance 
actions  minus  cannibalization.  However,  the  denominator  for 
Type  d  error' is  all  maintenance  actions,  including  cannibal¬ 
ization.  As  a  rule,  error  rates  are  determined  at  the  sub¬ 
system  level  by  work  center.  This  level  is  identified  by 
the  first  three  digits  of  the  Work  Unit  Code  (WUC) . 


Summary 

Three  kinds  of  maintenance  errors  have  been  defined.  Each 
has  been  examined  to  disclose  the  kinds  of  effects  it  pro¬ 
duces.  Effects  include  abort  flight,  repeat  function,  use 
spares  wastefully,  place  load  on  IMA,  and  risk  injury  or 
accident.  Methods  of  error  measurement  have  been  described 
and  shown  to  be  feasible,  given  real-world  data  of  the  qual¬ 
ity  provided  by  the  3-M  System. 


SYSTEM  DESIGN  AND  SUPPORT  FACTORS  AFFECTING  MAINTENANCE 

Maintenance  performance  at  the  organizational  level  is  in¬ 
fluenced  by  the  nature  of  the  aircraft  design  and  the  inter¬ 
play  of  its  various  support  elements.  In  this  section,  fac¬ 
tors  within  design  and  support  are  examined  for  their  gener¬ 
al  characteristics  and  direction  of  effect. 
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TABLE  9.  MALFUNCTION  CODES  INDICATING  DEFINITE  TYPE  d  ERROR 


CODE  DESCRIPTION 


086 

Improper  handling 

087 

Improper  identification 

246 

Improper  or  faulty  maintenance 

301 

Foreign  object  damage  (FOD) 

304 

FOD-Self  inducted 

651 

Air  in  system 

931 

Inadvertent  operation 
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TABLE  10.  MALFUNCTION  CODES  INDICATING  PROBABLE  TYPE  d  ERROR 


CODE 

093 

105 

106 

108 

127 

135 

410 

730 


DESCRIPTION 

Missing  part 

Loose  bolts,  nuts,  screws,  rivets,  fasteners,  etc. 
Missing  bolts,  nuts  or  screws 
Broken,  faulty,  or  missing  safety  wire  or  key 
Adjustment  or  alignment  improper 
Binding,  stuck,  jammed 
Lack  of,  or  improper  lube 

Loose 
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In  the  paragraphs  that  follow,  the  work  of  maintenance  is 
interfaced  with  the  performance  measures  introduced  earlier. 
After  that,  key  factors  in  design  and  support  are  examined 
for  their  effects  on  those  measures. 


Maintenance  Functions  and  Performance  Measures 

The  work  of  maintenance  is  best  expressed  in  terms  of 
maintenance  functions,  e.g.,  troubleshoot,  remove/ replace , 
service,  and  so  on.  At  the  organizational  level,  mainte¬ 
nance  functions  may  be  sorted  into  two  general  categories, 
as  shown  in  Table  11.  The  functions  in  one  category  are 
concerned  with  assessing  system  status.  The  functions  in 
the  other  category  are  concerned  with  correcting  or 
preserving  system  condition.  The  functions  thus  represented 
cover  both  corrective  and  preventive  maintenance. 

Functions  for  assessing  system  status  are  information- 
oriented  in  nature.  They  consist  of  troubleshooting  and 
checkout,  within  which  reside  lower-level  tasks  including 
operate,  test,  and  inspect.  Normally,  decision-oriented 
tasks  apply  at  the  system  level.  Their  object  is  to  obtain 
data  needed  to  answer  questions  about  the  condition  of  the 
system. 

Functions  for  correcting  or  preserving  system  condition  are 
action-oriented  in  nature.  They  consist  of  remove/ instal 1 , 
repair- in-place,  and  service.  Repair- in-pl ace  includes 
lower-level  tasks  such  as  adjust,  align,  and  replace  piece 
parts.  Service  includes  tasks  such  as  clean  and  lubricate. 
Normally,  action-oriented  tasks  apply  at  the  unit  level. 
Their  object  is  to  bring  the  system  into  compliance  with 
established  criteria  by  acting  on  units  within  the  system. 

Using  the  maintenance  performance  measures  defined  earlier, 
the  following  steps  can  be  taken  relative  to  effectiveness 
evaluation : 

•  The  effectiveness  of  maintenance  functions  for  assess¬ 
ing  system  status  can  be  measured  in  terms  of  the  Type 
I  error  rate. 

•  The  effectiveness  of  maintenance  functions  for  correct¬ 
ing  or  preserving  system  condition  can  be  measured  by 
the  Type  Il/d  error  rate. 

•  The  effectiveness  of  all  maintenance  functions  can  be 
measured  by  elapsed  (active)  maintenance  time. 

With  this  connection  made,  we  can  now  look  at  system  design 
and  support,  and  examine  the  effects  of  each  on  maintenance 
performance . 
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TABLE  11.  MAINTENANCE  FUNCTIONS  SORTED  BY  PURPOSE 


PURPOSE 

FUNCTION 

DESCRIPTION 

To  assess  status 
of  system 

Troubleshoot 

The  process  of  isolating  a  system  problem 
to  a  removable  unit  or  to  a  repairable  item. 

Checkout 

The  process  of  determining  whether  a 
system  is  in  an  operational  condition. 

(Troubleshoot  and  Checkout  include 
other  tasks  such  as  operate,  test,  and 
inspect.) 

To  correct  or 
preserve  condition 
of  system 

Remove/Install 

Repair-in-Place 

The  process  of  taking  out  a  unit  and  putting 
another  unit  in  its  place. 

The  process  of  correcting  a  malfunction 
without  replacing  a  unit. 

(Repair-in-Place  includes  other  tasks  such 
as  adjust,  align,  and  replace  piece  parts.) 

Service 

The  process  of  replenishing  consumables 
or  restoring  the  system/equipment. 

(Service  includes  other  tasks  such  as 
lubricate.) 
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System  Design  Factors  Affecting  Maintenance 

System  design  is  the  physical  means  through  which  the  engi¬ 
neer  seeks  to  promote  maintenance  effectiveness  by  reducing 
error  rate,  elapsed  maintenance  time,  and  maintenance  work¬ 
loads.  System  design  factors  affecting  maintenance  are  sum¬ 
marized  in  Table  12  and  explained  further  in  the  paragraphs 
that  follow. 


Design  Factors  Affecting  Assessment  of  System  Status 

System  status  can  be  assessed  by  using  symptoms  and  cues. 
Symptoms  are  system  outputs  that  fall  beyond  allowable 
operating  limits.  Cues  are  bits  of  information  concerning 
the  relationships  among  the  constituent  parts  of  the  system. 
Cues  help  identify  the  cause  of  system  failures  in  terms  of 
parts  not  performing  as  designed. 

(  As  a  general  rule,  symptoms  are  relatively  easy  to  define 

and  are  readily  available  for  observation.  The  problem  for 
r  the  technician  in  status  assessment  is  recognizing  and 

i  interpreting  the  cues.  The  problem  for  the  designer  is  to 

assure  that  cues  are  furnished  in  a  usable  form. 

j  Cues  indicating  system  status  occur  on  at  least  three  points 

i  of  a  continuum.  At  one  end  are  natural  cues.  At  the  other 

end  are  processed  cues.  Between  these  extremes  are  cue 
r  measurement  provision! . 

Natural  cues  are  most  typically  found  in  mechanical  systems 
where  the  parts  relate  to  each  other  in  concrete  and 
observable  ways.  Parts  move  in  predictable  patterns;  they 
become  worn,  loose,  misaligned,  and  broken;  they  react  to 
heat  and  pressure;  they  make  noises.  Natural  cues  correlate 
highly  with  system  output,  thus  providing  valuable  infor¬ 
mation  with  little  help  from  the  system  designer.  Natural 
.  cues  can  be  identified  by  the  technician  by  a  relatively 

simple  inspection. 

v  Processed  cues  are  most  typically  found  in  electronic 

systems,  where  relationships  among  the  parts  are  not  readily 
observable.  The  designer  must  provide  for  sensing,  measure¬ 
ment,  and  indication.  Processed  cues  may  be  relatively 
simple,  such  as  flight  indicator  malfunction  flaqs,  or 
r  extremely  complex,  such  as  those  driven  by  digital  com¬ 

puters.  If  correctly  designed,  processed  cues  demand  from 
the  technician  little  more  than  acknowledgement  of  final 
results . 
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TABLE  12.  SYSTEM  DESIGN  FACTORS  SORTED  BY  MAINTENANCE  PURPOSE 


MAINTENANCE 

PURPOSE 

MAINTENANCE 

FUNCTION 

SYSTEM  DESIGN  FACTORS 

MEASURE 

To  assess  status 
of  system 

Troubleshoot 

Checkout 

Type  1  Error 

Rate 

Elapsed  Maintenance 
Time* 

Provision  of: 

Status  information 

Action  ease  (on  test 
equipment) 

To  correct  or 
preserve  condition 

Remove/lnstall 

Repair-in-Place 

Provision  of: 

of  system 

! 

Service 

Type  il/d  Error 

Rate 

Elapsed  Maintenance 
Time* 

Physical  access 

Visual  access 

Action  accuracy 

Action  ease 

Secondary  damage 
prevention 

Handling  ease 

Unit,  feature  identification 

Procedural  information 

Personal  safety 

None 

None 

Mean  Time  Between 
Maintenance  Actions 

Reliability 

*  Elapsed  maintenance  time  is  time  spent  actually  doing 
the  work  (active  maintenance  time).  It  does  not  take  into 
account  the  effects  of  multiple-man  work  crews. 


36 


203C.81.6 


NADC-7 92 18-60 


Cue  measurement  provisions  are  needed  when  natural  cues  do 
not  suffice  and  processed  cues  are  not  feasible.  Examples 
of  cue  measurement  provisions  are  electrical  test  points  and 
wear  bars  on  tires.  Cue  measurement  provisions  must  be 
provided  by  the  designer.  By  definition,  such  provisions 
require  action  from  the  technician  in  the  form  of  testing 
and  inspection. 

Systems  that  provide  natural  cues  have  little  adverse  effect 
on  the  maintenance  error  rate  but  inadequacies  may  increase 
elapsed  maintenance  times.  Natural  cues  require  human  judg¬ 
ment  for  correct  interpretation.  Where  experienced  judgment 
is  lacking,  cues  are  often  misinterpreted  initially,  then 
corrected  later  in  the  same  maintenance  action.  The  penalty 
in  such  cases  is  an  excess  of  repair  time  rather  than  un¬ 
detected  errors. 

Systems  that  provide  processed  cues  may  have  widely  variable 
effects  on  the  maintenance  error  rate.  To  produce  maximum 
reduction  of  error  rate,  processed  cues  must  have  high  val¬ 
idity  (in  terms  of  their  correlation  with  system  malfunc¬ 
tions)  and  low  ambiguity  in  displayed  results.  Further,  to 
the  extent  that  the  technician  is  called  upon  to  resolve  the 
results  when  they  are  ambiguous,  the  technician  must  be 
given  enough  information  to  perform  that  role  effectively. 
Systems  dependent  on  processed  cues  not  meeting  these 
requirements  tend  to  exhibit  high  maintenance  error  rates. 

One  source  of  such  error  is  incomplete  provision  for  meas¬ 
urement  opportunities.  Another  source  is  excessive  demand 
for  reasoning  by  the  technician.  Either  type  of  hardware 
deficiency  can  create  problems.  Therefore,  both  must  be 
given  close  attention  by  the  prime  equipment  designer. 

Provisions  for  cue  measurement  often  necessitate  the  use  of 
support  equipment.  In  such  cases,  the  design  of  the  support 
equipment  can  have  a  bearing  on  maintenance  effectiveness. 
The  support  equipment  must  provide  for  ease  of  action  (as 
discussed  later  in  this  section) .  Where  ease  of  action  is 
not  promoted ,  the  most  typical  consequence  is  an  increase  in 
elapsed  maintenance  time.  Maintenance  errors  seldom  result, 
unless  the  equipment  is  so  difficult  to  use  that  the  tech¬ 
nician  attempts  to  do  without  it. 


Design  Factors  Affecting  Correction  or  Preservation  of 
System  Condition 

Design  factors  affecting  correction  or  preservation  of  sys¬ 
tem  conditions  are  covered  quite  well  in  MIL-STD-1472B , 
Section  5.9,  Design  for  Maintainability.  For  purposes  of 
convenience,  these  factors  are  represented  here  under  a 
small  number  of  key  headings: 
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•  Physical  access 

•  Action  accuracy 

•  Action  ease 

Each  heading  is  discussed  briefly  in  the  paragraphs  that 
follow.  Examples  are  given  from  MIL-STD-1472B. 

Physical  Access.  Physical  access  concerns  the  relative  lo- 
cation  of  equipment.  It  is  important  in  all  maintenance 
tasks  requiring  movement  or  manipulation  of  hardware  items. 
It  always  involves  physical  obstructions  in  one  form  or  an¬ 
other,  and  the  means  of  avoiding  them.  Where  such  obstruc¬ 
tions  are  removable  prior  to  reaching  the  object  unit,  they 
influence  maintenance  time.  Where  obstructions  are  not  re¬ 
movable,  they  may  influence  both  maintenance  time  and  the 
Type  d  error  rate.  Some  examples  of  access  provisions  in 
MIL-STD-1472B  are: 

5. 9. 4. 4  Rear  Access  Units  -  Sliding,  rotating  or  hinged 
units  to  which  rear  access  is  required  shall  be  free  to 
open  or  rotate  their  full  distance  and  remain  in  the  open 
position  without  being  supported  by  hand.  Rear  access 
shall  also  be  provided  to  plug  connectors  except  where 
precluded  by  any  other  operational  requirements. 

5. 9. 4. 5  Relative  Accessibility  -  In  determining  the  rela¬ 
tive  accessibility  of  units,  those  units  which  are  criti¬ 
cal  to  system  operation  and  which  require  rapid  mainte¬ 
nance  shall  be  most  accessible.  When  relative  criticality 
is  not  a  factor,  those  units  requiring  most  frequent  ac¬ 
cess  shall  be  most  accessible. 

5. 9. 4. 6  High-Failure-Rate  Items  -  The  physical  arrange¬ 
ment  of  units  and  components  should  be  such  that  high- 
failure-rate  items  will  be  accessible  for  replacement 
without  moving  non-failed  components  or  units.  Mechanical 
replacement  items  shall  be  removable  with  common  hand 
tools  and  simple  handling  equipment. 


Action  Accuracy.  Design  provisions  for  action  accuracy  rep- 
resent  direct  efforts  to  prevent  maintenance  errors.  They 
may  take  various  forms.  One  is  by  constraining  incorrect 
actions.  Another  is  by  providing  information  enabling  in¬ 
correct  actions  to  be  avoided  or  corrected.  Action  accuracy 
provisions  may  influence  both  the  Type  d  error  rate  and 
maintenance  time.  Some  examples  of  action  accuracy  pro¬ 
visions  in  MIL-STD-1472B  are: 

5.9.12.4  Alignment  -  Guide  pins  or  their  equivalent  shall 
be  provided  to  assist  in  alignment  during  mountinq,  par¬ 
ticularly  on  modules  that  are  connectors  themselves. 
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5.9.12.5  Coding  -  All  replaceable  items  shall  be  coded 
(i.e,  keye3")  so  that  it  will  be  physically  impossible  to 
insert  a  wrong  item.  Coding  by  such  means  as  color  or  la¬ 
bels  shall  identify  the  correct  item  and  its  proper  orien¬ 
tation  for  replacement. 

5. 9. 8. 2  Instructions  -  If  the  method  of  opening  a  cover 
is  not  obvious  from  the  construction  of  the  cover  itself, 
instructions  shall  be  permanently  displayed  on  the  outside 
of  the  cover. 


Action  Ease.  Design  provisions  on  behalf  of  action  ease 
seek  to  align  the  maintenance  task  with  the  known  limita¬ 
tions  and  capabilities  of  the  technician.  Such  provisions 
aim  to  reduce  the  demand  for  skill,  strength,  time,  and  even 
patience  on  the  job.  The  relative  degree  of  action  ease 
built  into  the  hardware  may  influence  both  the  Type  d  error 
rate  and  maintenance  time.  Some  examples  of  action  ease 
provisions  in  MIL-STD-1472B  are: 

5.9.10.8  Number  of  Turns  -  Fasteners  for  mounting  assem¬ 
blies  and  sub-assemblies  shall  require  only  one  complete 
turn,  provided  that  stress  and  load  considerations  are  not 
compromised.  When  bolts  are  required,  the  number  of  turns 
needed  to  tighten  and  loosen  them  shall  be  minimized. 

5.9.11.2  Extensions  -  Irregular,  fragile,  or  awkward  ex¬ 
tensions,  such  as  cables,  wave  guides,  hoses,  etc.,  shall 
be  designed  for  easy  removal  from  a  unit  before  the  unit 
is  handled. 

5. 9. 7. 4  Guides  -  Guides,  tracks,  and  stops  shall  be  pro¬ 
vided  as  necessary  to  facilitate  handling  and  to  prevent 
damage  to  units  and  components,  and  injury  to  personnel. 

5.9.11.4  Horizontal  Push  and  Pull  Forces  -  Manual  hori¬ 
zontal  push  and  pull  forces  required,  to  be  appl ied  in¬ 
itially  to  an  object  to  set  it  in  motion  or  to  be  sus¬ 
tained  over  a  period  of  time,  shall  not  exceed  the  values 
of  Table  XXII  [not  shown  here] ,  as  applicable. 


Hardware  Reliability 

One  other  design  factor  critical  to  the  maintainer  is  hard¬ 
ware  reliability.  While  reliability  changes  nothing  about 
the  performance  of  any  given  actio.n,  it  exerts  a  powerful 
influence  on  the  frequency  with  which  that  action  must  be 
taken.  The  higher  the  reliability,  the  greater  the  time  be¬ 
tween  maintenance  actions,  and  hence  the  lighter  the  work¬ 
load  on  affected  personnel. 
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Summary 

The  effects  of  system  design  on  maintenance  performance  are 
summarized  in  Table  13.  The  data  there  show  the  following 
relationships : 

•  Provisions  for  status  information  affect  the  Type  I  er¬ 
ror  rate  and  active  maintenance  time. 

•  Provisions  for  physical  access,  action  accuracy,  and 
action  ease  affect  the  Type  Il/d  error  rate  and  active 
maintenance  time. 

•  Hardware  reliability  affects  only  the  maintenance  work 
load. 

Status  information  and  action  ease  concern  troubleshooting. 
Physical  access,  action  accuracy,  and  action  ease  concern 
maintenance  actions  aimed  at  correcting  or  preserving  system 
condition.  Hardware  reliability  influences  the  frequency 
with  which  maintenance  actions  must  be  taken. 

System  Support  Factors 

It  has  been  shown  that  system  design  has  an  impact  on  all 
key  measures  of  maintenance  performance  at  the  organiza¬ 
tional  level.  In  the  paragraphs  that  follow,  system  support 
factors  are  considered  in  connection  with  these  same  meas¬ 
ures.  The  support  factors  involved  are  facilities  and 
equipment,  spares,  manning,  technical  information,  and 
maintenance  management. 


Facilities  and  Equipment 

Facilities  and  equipment  are  vital  to  maintenance,  but  as  a 
general  rule,  they  have  limited  influence  on  maintenance 
error  rates  or  active  maintenance  time.  They  also  have 
limited  influence  on  any  of  the  other  listed  support 
factors . 

Facilities  and  equipment  can,  however,  directly  affect  ad¬ 
ministrative  and  supply  downtime,  as  a  function  of  their 
availability  when  needed. 


Spares 

Spares  influence  maintenance  performance  in  a  manner  similar 
to  that  described  for  facilities  and  ecmipment.  They 
directly  affect  supply  downtime  only. 
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TABLE  13.  EFFECTS  OF  SYSTEM  DESIGN  FACTORS  ON  MAINTENANCE  PERFORMANCE 

P 

f 


SYSTEM  DESIGN 

FACTOR  EXERTING 

EFFECT 

MAINTENANCE  PERFORMANCE 
MEASURES  AFFECTED 

WORK  LOAD 
AFFECTED 

TYPE  1 

ERROR  RATE 

TYPE  ll/d 
ERROR  RATE 

MAINTENANCE 

TIME* 

STATUS  INFORMATION 

Yes 

No 

Yes 

No 

PHYSICAL  ACCESS 

No 

Yes 

Yes 

No 

ACTION  ACCURACY 

No 

Yes 

Yes 

No 

ACTION  EASE 

No 

Yes 

Yes 

No 

HARDWARE  RELIABILITY 

No 

No 

No 

Yes 

active  (elapsed)  maintenance  time 


f  • 

i 

i — 

| 
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Spares  have  no  effect  on  active  maintenance  time,  mainte¬ 
nance  error  rate,  or  any  of  the  other  listed  support 
factors . 


Manning 

Maintenance  manning  directly  affects  all  measures  of  mainte¬ 
nance  performance.  Technical  personnel  can  vary  widely  in 
both  quality  and  quantity.  Personnel  quality  influences 
performance  in  terms  of  error  rate  and  active  maintenance 
time.  Personnel  quantity  influences  administrative  downtime 
as  a  function  of  the  availability  of  personnel  when  needed. 

Manning  also  exerts  indirect  effects  on  maintenance  perform¬ 
ance.  Manning  quality  influences  the  demand  for  spares.  As 
the  need  for  spares  varies,  supplydown  time  varies  also. 


Technical  Information 

Technical  information  is  delivered  to  maintenance  personnel 
through  training  and  publications.  Training  seeks  to  place 
information  into  long-term  memory.  Publications  aim  to  pro¬ 
vide  a  reference  file  for  use  in  the  job  environment. 

Where  technical  information  is  incomplete,  inaccurate,  or 
low  in  usability  at  the  point  of  need,  it  can  increase  both 
the  error  rate  and  active  maintenance  time.  However,  tech¬ 
nical  information  has  no  effect  on  administrative  and  supply 
downtime . 

Technical  information  has  no  effect  on  facilities  and  equip¬ 
ment,  but  it  does  influence  spares,  manning,  and  management. 

Technical  information  affects  spares  by  altering  the  error 
rate.  It  affects  manning  by  influencing  personnel  perform¬ 
ance,  quality,  and  flexibility.  It  affects  management  by 
altering  the  number  of  options  available  to  supervisors. 
These  options  concern  technician  development  and  work 
assignment . 


Maintenance  Management 

Maintenance  management  covers  both  supervision  and  higher- 
level  planning.  Technical  supervision  can  directly  influ¬ 
ence  the  error  rate  and  active  maintenance  time,  through 
day-by-day  methods  of  motivating,  developing,  and  disciplin¬ 
ing  personnel . 
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Higher-level  planning  can  directly  affect  the  error  rate  and 
administrative  downtime  through  the  application  of  priori¬ 
ties  in  scheduling. 

Like  technical  information,  maintenance  management  has  no 
effect  on  facilities  and  equipment  but  does  influence  all 
other  listed  maintenance  support  factors. 

Management  affects  the  availability  of  spares  by  its  han¬ 
dling  of  the  diagnostic  function.  The  demand  for  spares  is 
highly  sensitive  to  diagnostic  accuracy.  Trial  and  error 
methods  tend  to  be  wasteful  of  spares.  Where  management 
priorities  emphasize  work  speed  rather  than  accuracy,  the 
problem  is  aggravated  further. 

Management  affects  manning  in  maintenance  in  the  same  way  as 
is  seen  elsewhere.  Supervisory  quality  influences  personnel 
motivation  and  development.  These  factors,  in  turn,  impact 
the  error  rate,  active  maintenance  time,  and  willingness  to 
remain  in  service. 

Management  affects  technical  information  by  establishing 
rules  concerning  its  use.  Management  can  encourage  or  dis¬ 
courage  technicians  from  referring  to  publications  for  help; 
it  can  use  or  ignore  information  designed  to  promote  person¬ 
nel  development.  Where  management  properly  exploits  high- 
quality  information,  the  error  rate  declines,  active  mainte¬ 
nance  time  decreases,  and  low-skill  personnel  become  avail¬ 
able  for  a  wider  range  of  assignments. 


Summary 

The  effects  of  the  various  support  factors  on  maintenance 
performance  are  summarized  in  Table  14.  The  data  there 
indicate  that  the  support  factors  with  the  greatest  influ¬ 
ence  are  maintenance  management  and  technical  information. 
The  factors  with  the  least  influence  are  facilities  and 
support  equipment,  and  spares.  Manning  has  a  strong  direct 
influence  but  has  little  effect  on  other  support  factors. 

Facilities  and  support  equipment,  and  spares  have  little  or 
no  effect  on  the  error’ rate  or  active  maintenance  time  but 
do  influence  other  downtime. 

Manning,  technical  information,  and  maintenance  management 
influence  the  error  rate,  active  maintenance  time,  and  ad¬ 
ministrative  and  supply  downtime.  Such  downtime  includes 
time  waiting  for  spares,  equipment,  and  personnel. 


43 


203C.81.6 


NADC-79 218-60 


TABLE  14.  EFFECTS  OF  SYSTEM  SUPPORT  FACTORS  ON  MAINTENANCE  PERFORMANCE 


FACTORS  AFFECTED 

SYSTEM  SUPPORT 

FACTOR  EXERTING 

EFFECT 

MAINTENANCE  PERFORMANCE 
MEASURES  AFFECTED 

Q 

Z 

< 

oo  !r 

LU  z 

PS 

00 

<3 

Z 

Z 

o 

11 

U1  L_ 

H 

ERROR 

RATE 

ELAPSED 

MAINT. 

TIME 

* 

OTHER 

DOWN 

TIME 
*  * 

— 

-J  CL 

O  D 
<  O 

LL  Lit 

LU 

cn 

< 

Q. 

CO 

Z 

z 

< 

5 

52 

£5 

z 

<  < 

2  2 

FACILITIES  AND 

EQUIPMENT 

NO 

NO 

YES 

- 

NO 

NO 

NO 

NO 

SPARES 

NO 

NO 

YES 

NO 

- 

NO 

NO 

NO 

MANNING 

YES 

YES 

YES 

NO 

YES 

- 

NO  ! 

NO 

TECHNICAL  INFORMATION 

YES 

YES 

NO 

NO 

YES 

YES 

- 

YES 

MAINTENANCE 

MANAGEMENT 

YES 

YES 

YES 

NO 

YES 

YES 

YES 

- 

KIND  OF  EFFECT 

DIRECT 

INDIRECT 

*  Elapsed  maintenance  time  is  time  spent  actually  doing  the  work 

(active  maintenance  time).  It  does  not  take  into  account  the  effects 
of  multiple-man  work  crews. 

**  Other  downtime  refers  to  delays  in  work  completion,  due  to  any 
reason  other  than  elapsed  maintenance  time. 
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SECTION  II 

QUANTITATIVE  ASPECTS  OF  THE  PROBLEM 


The  design- for-maintainer  problem  is  explored  quantitatively 
under  the  following  headings: 

•  The  AMES  Model 

•  Other  Research  on  Link  Between  Maintenance 
Effectiveness  and  System  Readiness 


THE  AMES  MODEL 

Several  key  points  have  been  made  relative  to  maintenance 
effectiveness.  One  is  that  maintenance  performance  is  in¬ 
fluenced  by  system  design  and  support  factors.  Another  is 
that  maintenance  performance  can  be  measured  in  terms  of 
error  rate  and  maintenance  time.  A  third  point  is  that  a 
quantitative  connection  can  be  made  between  maintenance 
effectiveness  and  system  readiness. 

From  these  facts ,  a  linkage  can  be  constructed,  from  design 
for  the  maintainer,  through  measurements  of  maintenance 
effectiveness,  to  projections  of  system  readiness. 

The  linkage  referred  to  here  has  huge  implications  for 
weapon  system  planning.  What  it  means  is  that  maintenance 
effectiveness  correlates  with  system  readiness  well  enough 
to  function  as  a  predictor.  Therefore,  any  proposed  im¬ 
provement  in  maintenance  effectiveness  can  be  evaluated  in 
advance  by  projecting  a  corresponding  impact  on  system  read¬ 
iness..  The  potential  thus  exists  for  conducting  cost- 
effectiveness  studies  in  advance  of  funding  commitments. 

Given  the  validity  of  this  statement,  a  natural  question 
arises:  how  can  the  linkage  be  exercised  efficiently,  in 
the  face  of  a  maintenance  environment  containing  so  many 
interacting  variables?  The  answer  is  simulation  modeling. 
Aircraft  Maintenance  Effectiveness  Simulation  (AMES)  is  a 
model  designed  specifically  for  this  purpose. 


General  Characteristics 

The  AMES  Model  is  a  computer  program  representing  the  oper¬ 
ating  and  maintenance  environment  of  a  designated  squadron 
of  aircraft  for  a  given  period  of  time.  Its  purpose  is  to 
permit  the  economical  study  of  the  various  personnel/human 
factors  in  maintenance  that  contribute  to  aircraft 
operational  readiness. 
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The  operating  and  maintenance  environments  are  expressed  as 
a  set  of  related  functions,  such  as  Dispatch  Aircraft,  Fly 
Mission,  Conduct  Post-Flight  Inspection,  and  Provide  Spare 
Parts.  The  model  exercises  each  function  and  its  logical 
relationships  with  other  functions.  Information  'flows  from 
function  to  function  as  real-life  events  are  simulated. 

Thus,  aircraft  are  dispatched  in  accordance  with  a  set 
flight  schedule.  In  the  course  of  their  missions  relevant 
subsystems  are  operated.  Some  subsystems  malfunction,  as  a 
reflection  of  component  failure  rates.  Others  perform  as 
intended . 

Upon  returning  to  the  carrier,  the  (simulated)  aircraft  are 
serviced  in  the  normal  manner.  Technicians  are  assigned 
from  duty  rosters  in  established  work  centers.  They  perform 
troubleshooting  and  corrective  maintenance  on  the  malfunc¬ 
tioned  subsystems.  These  actions  consume  time  analogous  to 
that  shown  in  the  3-M  records.  In  some  cases,  the  techni¬ 
cians  commit  errors. 

The  effects  of  time  consumption  are  treated  in  familiar 
ways.  While  working  on  one  job,  a  given  technician  is. not 
available  for  assignment  to  another.  In  addition,  until 
that  job  is  completed,  the  aircraft  is  considered  not  opera¬ 
tionally  ready.  As  the  workload  (for  a  personnel  specialty) 
increases,  queues  begin  to  form,  wherein  jobs  are  waiting 
for  people.  In  extreme  cases,  flights  are  scrubbed. 

The  effects  of  errors  are  treated  in  ways  that  show  interac¬ 
tion  with  other  key  variables.  A  troubleshooting  error,  for 
example,  is  "remembered"  and  allowed  to  be  discovered  during 
a  subsequent  pre- flight  inspection  or  in-flight  operation. 

If  rework  is  found  to  be  necessary,  it  is  handled  in  the 
same  manner  described  for  an  original  job. 

Detailed  records  are  kept  concerning  aircraft  status  and  the 
availability  of  all  the  resources  needed  for  proper  mainte¬ 
nance.  All  variables,  including  error  rate,  are  expressed 
quantitatively.  Reports  are  issued  tracking  operational 
readiness  and  other  selected  measures. 


Key  Particulars 

A  complete  description  of  AMES  is  presented  in  Aircraft 
Maintenance  Effectiveness  Simulation  (AMES)  Model;  Final 
Report  (NAVTRAEQUIPCSM  77-D-U028-1 )  .  Provided  here,  Tn  sum- 
mary  form,  are  descriptions  of  three  key  particulars: 
inputs  required,  outputs  required,  and  outputs  produced. 

The  model  s  treatment  of  variables  with  undetermined  values 
is  also  described. 
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Inputs  Required 

The  inputs  to  AMES  include  operating  schedules,  equipment 
item  reliability  values,  maintenance  manpower  levels,  time- 
to-repair  values,  and  personnel  error  rates.  All  such  data 
are  directly  available  from  existing  records.  A  listing  of 
prominent  variables  handled  by  the  model  appears  in 
Table  15.  These  inputs  are  best  described  in  terms  of  type 
and  form.  — £ — 


Types  of  Input  Data.  Six  types  of  input  data  are  used  by 
tfte  model.  They  are : 

•  subsystem  data 

•  component  data 

•  manpower  data 

•  planned  maintenance  data 

•  multipliers  data 

•  run  data 

Subsystem  data  refers  to  those  factors  required  .by  the  com- 
puter  to  simulate  each  subsystem  of  the  aircraft.  For  exam¬ 
ple,  CRITICALITY  expresses  the  probability  that  a  particular 
component  failure  in  the  subsystem  causes  the  aircraft  to  be 
NOR  (not  operationally  ready) . 

Component  data  refers  to  a  subsystem  combined  with  a  work 
center  (a  technical  maintenance  specialty  area).  As  can  be 
seen  in  Table  15,  fourteen  variables  comprise  a  definition 
for  each  component.  Many  of  these  have  already  been  defined 
for  the  reader. 

Manpower  data  refers  to  personnel  assigned  to  work  centers. 

A  work  center  identifies  a  group  of  specialized  personnel 
responsible  for  a  designated  area  of  aircraft  maintenance. 

Planned  maintenance  data  refers  to  maintenance  actions  that 
must  be  performed  on  a  regular  basis.  These  actions  are 
included  in  the  model  because  they  impose  a  significant 
workload  on  the  work  centers. 

The  model  identifies  the  different  planned  maintenance  re¬ 
quirements  . by  performance  intervals.  Each  planned  mainte¬ 
nance  requirement  is  divided  into  several  segments.  Each 
segment  requires  a  specified  amount  of  time  for  the  work  to 
be  performed.  A  segment  may  require  work  at  several 
centers. 

Multipliers  data  are  used  to  make  uniform  chanqes  of  human 
performance  variables.  For  each  of  the  components  or  sub¬ 
systems,  the  chosen  variable  is  multiplied  by  the  data  mul¬ 
tiplier  in  question.  A  data  multiplier  of  1.00  leaves 
values  unchanged.  For  example,  the  Type  I  error  rate  for 
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TABLE  15.  INPUTS  REQUIRED  BY  AMES  MODEL 


FORMS  OF 

INPUT  DATA 

INPUT 

VARIABLE 

TYPES  OF 

INPUT  DATA 

Integer  Variables 

Subsystem 

Component 

Work  Center 

Component 

Number  of  Work  Centers 

Manpower 

Number  of  Shifts 

Manpower 

Number  of  CDIs 

Manpower 

Number  of  Seniors 

Manpower 

Number  of  Juniors 

Manpower 

PM  Type 

Planned  Maintenance 

Number  of  Segments 

Planned  Maintenance 

Number  of  Work  Centers 

Planned  Maintenance 

PM  Work  Center 

Planned  Maintenance 

PM  Number  of  Men 

Planned  Maintenance 

Real  Variables 

Criticality* 

Subsystem 

Facilities* 

Subsystem 

RIP  Inventory* 

Component 

RR  Inventory* 

Component 

RR  Inventory  Part  Bad* 

Component 

Error  I* 

Component 

Error  ll/d* 

Component 

MTBF 

Component 

MTBND 

Component 

Mean  Technicians 

Component 

Length  of  Shift 

Manpower 

Segment  Time 

Planned  Maintenance 

Error  Type  1  Multiplier 

Multiplier 

Error  Type  ll/d  Multiplier 
Elapsed  Maintenance 

Multiplier 

Hours  Multiplier 

Parts  Bad  from  Supply 

Multiplier 

Multiplier 

Multiplier 

No  Defect  Multiplier 

Multiplier 

Compound  Variable 

Action  Taken* 

Component 

Distribution  Variables 

Facilities  Delay  Time* 

Subsystem 

RIP  Parts  Delivery  Time* 

Component 

RR  Parts  Delivery  Time* 

Component 

Elapsed  Maintenance  Hours* 

Component 

*  variables  treated  probabilistically 
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all  components  can  be  reduced  to  one-half  of  its  original 
value  by  setting  the  ERROR  TYPE  I  MULTIPLIER  to  0.5.  For 
each  variable,  a  data  multiplier  will  modify  the  associated 
variable  for  all  components  in  a  uniform  manner. 

Run  data  include  other  miscellaneous  entries  such  as  mission 
schedule ,  report  intervals,  RUN  DURATION,  and  traces.  Mis¬ 
sion  schedule  includes  a  MISSION  START  TIME  and  a  MISSION 
DURATION  for  each  mission.  The  report  intervals  are  varia¬ 
bles  that  adjust  the  time  between  reports.  RUN  DURATION  is 
the  duration  of  the  simulation  in  days.  Traces  are  used  for 
testing  and  validating  the  model. 


Forms  of  Input  Data.  To  be  used  by  the  model,  input  data 
must  be  expressed  In  one  of  the  following  forms: 

•  integer 

•  real  number 

•  compound  vari'able 

•  distribution 

Integer  variables  denote  quantity  or  identification.  They 
are  described  in  terms  of  whole  numbers.  For  example,  two 
senior  technicians  at  Work  Center  £. 

Real  Variables  are  used  on  single-dimensional  measures  that 
do  not  lend  themselves  to  treatment  as  integers.  For  exam¬ 
ple,  12.5  hours  as  the  mean  time  between  failures,  and  .34 
as  the  probability  of  a  Type  I  error  occurring. 

Compound  variables  are  used  in  situations  where  a  selection 
must  be  made  from  several  alternative  courses  of  action.  A 
typical  selection  situation  occurs  when  one  of  three  kinds 
of  maintenance  jobs  must  be  executed. 

Distribution  variables  are  used  in  situations  where  a  range 
of  values  may  apply,  each  with  its  own  probability  of  occur¬ 
rence.  Elapsed  maintenance  time  is  best  described  by  a 
distribution. 


Outputs  Produced 

The  outputs  produced  by  the  model  consist  of  permanent  re¬ 
ports  and  processing  traces.  Permanent  reports  are  docu¬ 
ments  designed  for  use  by  squadron  commanders  and  mainte¬ 
nance  managers.  They  cover  operational  variables  such  as 
system  readiness,  maintenance  manhours  per  flight  hour, 
missions  completed,  and  reasons  for  delay.  Traces  are 
designed  for  use  by  personnel  running  the  model.  They 
concern  the  detailed  workings  of  the  program. 
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Permanent  Reports.  The  following  reports  are  available 
whenever  the  model  is  run: 

•  Monthly  Readiness  Reporting  Status  (RRS) 
report  by  aircraft 

•  History  of  RRS 

•  Manhour  summary 

•  Additional  reports 

Monthly  Readiness  Reporting  Status  (RRS)  report  by  aircraft 
cover  such  statistics  as:  hours  operationally  ready,  hours 
full  systems  capable,  total  flight  hours,  and  numbers  of 
sorties  flown.  In  addition,  aircraft  unavailability  (NOR) 
and  Reduced  Material  Conditions  (RMC)  are  categorized  in 
terms  of:  unscheduled  maintenance  hours,  hours  awaiting 
maintenance,  and  not  fully  equipped  (NFE) . 

History  of  RRS  (12  month)  contains  RRS  percentages,  total 
flights,  and  total  fl ight  hours  during  the  period,  as  well 
as  aircraft  utilization.  The  percentages  are  taken  from  the 
bottom  of  the  RRS  report  for  each  of  the  preceding  12  re¬ 
ports.  Aircraft  utilization  is  the  average  number  of  flight 
hours  per  aircraft  during  the  period. 

Manhour  summary  records  manhour  statistics  by  manhours  of 
maintenance  time  required.  The  manhours  and  manhours  per 
flight  hour  (MMH/FH)  are  obtained  from  3-M  records.  The 
manhour  summary  of  the  model  does  not  go  into  the  same  de¬ 
tail  with  Preventive  Maintenance  (PM)  as  do  the  3-M  reports. 
PM  is  not  considered  in  detail  because  it  is  not  central  to 
the  model. 

The  manhour  summary  also  includes  miscellaneous  information 
applicable  to  model  operation:  number  of  Il/d  errors;  num¬ 
ber  of  missions  flown,  scrubbed,  or  deferred;  and  the  aver¬ 
ages  of  the  percent  of  time  OR  and  the  percent  of  missions 
flown . 

Additional  reports  cover  the  total  number  of  jobs  by  type, 
total  elapsed  maintenance  time  (EMT) ,  total  flights,  total 
flight  hours,  and  the  location  of  each  aircraft  at  the  time 
of  the  report.  These  reports  are  useful  for  comparison  when 
error  rates  are  changed. 

Processing  Traces.  Processing  traces  are  outputs  selected 
at  the  time  the  simulation  is  begun.  They  were  originally 
built  into  the  program  as  debugging  tools.  Such  traces  are 
useful  for  examining  in  detail  the  workings  of  the  model. 
They  include  such  specific  items  as  takeoffs,  landings,  and 
mission  deferrals.  The  model  offers  more  than  30  traces 


50 


203C.81.6 


NADC-7 92 18-60 


covering  detailed  information  on  inputs,  corrective  main¬ 
tenance  jobs,  inspection  reports,  component  failures,  etc. 
Traces  are  essential  for  model  testing. 


Treatment  of  Variables  with  Undetermined  Values 

Many  of  the  variables  dealt  with  by  the  model  have  specific 
values  that  are  unknown  at  the  time  the  model  is  run.  One 
example  would  be  whether  or  not  a  Type  I  error  should  be 
simulated  in  a  given  situation.  Another  would  be  the  number 
of  elapsed  maintenance  hours  (to  be  simulated)  on  a  given 
job.  These  and  other  variables  with  undetermined  values  are 
marked  with  an  asterisk  in  Table  15. 

As  shown  in  the  footnote  of  Table  15,  these  variables  are 
treated  probabilistically  by  the  model.  The  method  used  is 
the  Monte  Carlo  technique.  A  brief  description  of  this 
technique  is  given  below. 

In  preparation  for  a  model  run,  a  table  is  constructed  for 
each  variable  to  be  handled  probabilistically.  The  table 
contains  all  possible  values  of  the  variable.  A  listing  of 
random  numbers  is  then  provided  along  with  a  decision  rule 
connecting  the  list  to  the  table.  During  the  run,  when  the 
need  for  a  value  occurs,  a  number  is  selected  randomly  from 
the  list.  When  the  decision  rule  is  applied,  the  random 
number  determines  the  value  to  be  drawn  from  the  table. 

As  an  illustration,  consider  the  question  of  when  to  simu¬ 
late  a  component  failure.  Assume  that  the  real-life  failure 
rate  is  such  that,  in  any  given  hour,  the  probability  of  the 
component's  failure  is  15  percent.  The  table  of  possible 
values  is  constructed  to  show  two  conditions:  Failed  and 
Not  Failed.  The  list  of  random  numbers  is  made  to  contain 
all  percent  values  between  zero  and  one.  The  decision  rule 
dictates  that  all  numbers  from  zero  to  .15  will  represent 
the  Failed  condition,  while  those  above  .15  will  represent 
the  Not  Failed  condition.  At  the  time  of  need,  a  number  is 
selected  at  random  from  the  list.  The  size  of  the  number 
determines  which  of  the  two  conditions  is  to  be  invoked  by 
the  model. 


Typical  Application 


In  a  typical  application,  the  AMES  Model  would  be  used  to 
measure  the  relationship  between  a  particular  aspect  of 
maintenance  effectiveness  and  a  system  effectiveness  param¬ 
eter  such  as  missions  completed. 

The  3-M  data  bank  for  the  aircraft  of  concern  could  be  in¬ 
terrogated  to  fill  the  input  requirements  of  the  model.  The 
model  would  be  run  several  times,  to  stabilize  the  relation¬ 
ship  between  those  inputs  and  the  output  of  interest. 
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Assume  the  aspect  of  maintenance  effectiveness  to  be  studied 
is  the  Type  I  error  rate  under  the  influence  of  a  postulated 
design  change  affecting  troubleshooting.  An  estimate  would 
be  made  of  improvement  in  the  Type  I  error  rate  that  would 
result  if  the  design  change  were  implemented. 

A  revised  set  of  error  rates  would  be  entered,  relative  to 
every  subsystem  to  be  impacted  by  the  design  change.  The 
model  would  be  run  again.  The  resulting  expression  of 
missions  completed  could  then  be  compared  to  the  pre-change 
value,  and  the  extent  of  the  gain  easily  determined. 


OTHER  RESEARCH  ON  LINK  BETWEEN  MAINTENANCE  EFFECTIVENESS  AND 
SYSTEM  READINESS 

The  relationship  between  maintenance  effectiveness  and 
system  readiness  has  been  explored  many  times.  Because  of 
the  complexity  of  the  maintenance/operations  environments, 
with  their  multitude  of  interacting  variables,  all  studies 
of  any  consequence  have  made  use  of  computer  technology. 

Most  typically,  they  have  taken  the  form  of  simulation 
models.  Six  of  the  most  representative  efforts  are  reviewed 
in  this  section.  They  are: 

•  Logistics  Composite  Model  (L-COM) 

•  Validated  Aircraft  Logistics  Utilization  Evacuation 
(VALUE  IV) 

•  Carrier  Aircraft  Integrated  Requirements  (CAIR) 

•  Comprehensive  Aircraft  Support  Effectiveness  Evaluation 
(CASEE) 

•  Ship  Simulation  Model 

•  Military  Manpower  vs.  Hardware  Procurement  (HARDMAN)- 


Logistics  Composite  Model  (L-COM) 

Logistics  models  have  been  used  extensively  in  the  past  for 
maintenance  management.  L-COM  (Fisher,  1968)  is  a  computer- 
based  model  for  simulating  a  composite  of  operations  and 
support  functions  at  an  Air  Force  base.  L-COM  simulates 
sortie  requirements,  aircraft  flights,  servicing  task  accom¬ 
plishments,  equipment  malfunctions,  repair  and  maintenance 
operations,  and  resource  utilization.  L-COM  enables  a  de¬ 
termination  of  the  "best  mix"  of  resource  elements  for  in¬ 
creased  cost  effectiveness  in  system  support. 


52 


203C.81.6 


NADC-79218-60 


The  model  consists  of  three  main  programs:  a  pre- processor , 
a  simulation  program,  and  a  post- processor . 

Extensions  and  refinements  to  L-COM  have  incorporated  a 
repair-level  decision  model,  multibase  operations,  weather 
effects,  an  equipment  cannibalization  model,  and  various 
other  representations  of  compatible  and  conflicting 
maintenance  operations. 


Validated  Aircraft  Logistics  Utilization  Evaluation 

7VKE0E  'TV)' - 

VALUE  IV  (Systems  Analysis  and  Engineering  Department,  NADC, 
1969)  allows  for  concurrent  exercising  of  multiple  squadrons 
of  different  types  of  aircraft.  Each  has  its  own  operating 
philosophy,  manpower  requirements,  shift  assignments,  and 
aircraft  characteristics.  The  model  provides  a  complete 
profile  of  flight  operations,  maintenance  activities,  and 
other  related  operational  factors  by  aircraft  type  and 
squadron.  It  is  capable  of  differential  analysis  of  the 
aircraft,  the  carrier,  and  the  support  system.  Many  de¬ 
cision  points  within  the  model,  such  as  repair  times,  main¬ 
tenance  action  frequency  by  inspection,  delays  due  to  ground 
support  equipment,  etc.,  are  obtained  by  employing  Monte 
Carlo  techniques.  It  was  validated  for  several  aircraft 
types  using  the  Navy's  3-M  data  system. 


Carrier  Aircraft  Integrated  Requirements  (CAIR) 

The  CAIR  simulation  model  (Engineering  Management  Department 
USN ,  1975),  is  designed  to  assess  Level  of  Repair  (LOR) 
assignments  for  carrier-based  airborne  equipment  and  to  de¬ 
termine  their  impact  on  operational  effectiveness.  The  var¬ 
ious  segments  of  the  CAIR  analysis  provide  information  con¬ 
cerning  the  relative  costs  of  maintenance-support  alterna¬ 
tives  and  the  effectiveness  of  each  support  alternative. 

The  model  may  be  used  to  specify  LOR  assignments  that  mini¬ 
mize  life-cycle  logistics  support  costs.  It  may  also  eval¬ 
uate  the  life-cycle  logistics  support  cost  for  a  prescribed 
set  of  assignments  of  the  components,  e.g.,  avionic 
equipment . 

In  order  to  exercise  the  model,  quantitative  data  are  re¬ 
quired  for  each  assembly  operational  site,  support  equipment 
type,  and  type  of  manpower.  By  using  this  data  in  associa¬ 
tion  with  LOR  assignments,  the  simulator  can  predict  the 
ability  of  the  carrier  to  meet  its  demand  for  missions.  The 
simulator  can  also  identify  those  personnel,  qround  support 
equipment,  and  spares  that  may  cause  bottlenecks  in  the 
maintenance  support  system. 
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Comprehensive  Aircraft  Support  Effectiveness  Evaluation 
(CASEE) 

THE  CASEE  model  (U.S.  Naval  Air  Systems  Command,  1977)  is 
primarily  intended  as  a  system  analysis  tool  to  help  evalu¬ 
ate  various  factors  that  influence  fleet  readiness  and  mis¬ 
sion  availability  in  a  carrier  operational  environment.  The 
model  considers  such  factors  as  reliability  and  maintaina¬ 
bility  of  the  aircraft  systems  and  subsystems,  spare  parts 
availability,  and  maintenance  facility  availability.  The 
model  also  considers  in  some  detail  the  manpower  reauire- 
ments  for  both  organizational  and  intermediate  maintenance 
levels . 

By  exercising  the  model  with  appropriate  inputs  (Navy  3-M 
Data,  fleet  records,  aircraft  technical  manuals,  etc.)  , 

CASEE  can  be  used  as  a  management  tool  to  help  managers  make 
analytical  decisions  based  on  quantified  information.  These 
decisions  can  revolve  around  such  areas  as  equipment  design 
improvements,  reliability  and  maintainability  optimization, 
support  planning,  fleet  readiness  improvement,  and  life- 
cycle  cost  studies.  In  addition,  CASEE  can  be  used  to  de¬ 
termine  trade-offs  between  spares  provisioning  and  fleet 
read iness . 

CASEE  is  capable  of  simulating  up  to  127  individual  aircraft 
of  various  types  from  as  many  as  10  different  organizational 
units.  The  current  version  of  CASEE  has  been  numerically 
validated  against  real-world  Naval  aircraft  operations  and 
support . 


Ship  Simulation  Model 


The  total  Ship  Simulation  Model  was  developed  in  the  mid- 
1960  's  to  provide  the  Navy  with  a  means  of  performing  per¬ 
sonnel  research  at  the  total  systems  level .  Many  of  the 
development  procedures  first  established  in  the  Ship  Simu¬ 
lation  Model  have  been  employed  in  subsequent  models  such  as 
AMES. 

Once  the  objectives  and  constraints  are  generated  and  meas¬ 
ures  of  system  performance  identified,  a  functions  analysis 
is  conducted.  The  functions  analysis  is  essentially  a  proc¬ 
ess  of  tracing  the  demands  placed  on  a  ship  through  the 
various  functions  which  the  ship  performs.  A  functional 
representation  based  on  this  analysis  is  coupled  with  de¬ 
tailed  Navy  Doctrine  Procedures  to  develop  functional  speci¬ 
fications  for  the  simulation  model.  Data  from  representa¬ 
tive  maintenance  actions  are  processed  through  the  model, 
and  a  rank  order  analysis  of  equipment  is  performed. 
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Outputs  from  the  model  provide  a  summary  of  corrective  and 
planned  maintenance  activities.  Problems  in  system  readi¬ 
ness  can  be  traced  to  specific  equipment  to  determine  the 
causes  of  equipment  downtime.  The  model  also  produces  a 
summary  of  the  personnel  training  readiness  resulting  from 
the  simulation  run. 

Potential  applications  of  the  ship  simulation  model  range 
from  establishing  the  optimum  mix  of  personnel  and  evalua¬ 
tion  of  proposed  automation  concepts,  to  identification  of 
system  problems. 

The  model  was  originally  designed  and  validated  using  the 
DDG-2  class  Guided  Missile  Destroyer  as  its  object  system. 


Military  Manpower  vs.  Hardware  Procurement  (HARDMAN) 

HARDMAN  is  not  a  computer  simulation  model,  but  rather  a 
methodological  approach  to  maintenance  manpower  require¬ 
ments.  The  HARDMAN  methodology  (Dynamics  Research  Corp., 
1979)  is  used  to  address  manpower,  personnel,  and  training 
requirements  in  the  weapon  system  acquisition  process 
(WSAP) .  It  is  a  relatively  new  method  based  on  3-M  data 
collected  from  proposed  weapon  systems  (known  as  reference 
systems) .  These  data  include  information  that  characterizes 
the  equipment,  maintenance  concept,  operator  tasks,  and  man¬ 
power  requirements  of  the  system.  This  is  in  terms  of  speed 
and  certain  accuracy  measures. 

The  HARDMAN  system  indicates  where  maintenance  manhour  re¬ 
quirements  can  be  reduced  without  a  loss  of  overall  system 
effectiveness.  This  results  in  lower  maintenance  personnel 
requirements  for  the  baseline  weapons  system.  Data  are  pro¬ 
vided  on  potential  improvements  in  system  reliability  and 
aircraft  availability. 

The  concept  of  maintenance  errors  and  reduction  of  mainte¬ 
nance  error  rates  is  not  really  considered  when  assessing 
the  effect  of  a  proposed  systems  design  change.  The  HARDMAN 
methodology  was  initially  applied  to  the  Shipboard 
Intermediate  Range  Combat  System  (SIRCS) . 


Summary 

A  review  has  been  made  of  six  major  efforts  to  explore  the 
connection  between  maintenance  effectiveness  and  system 
readiness.  The  following  conclusions  can  be  drawn  regarding 
these  studies. 
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First,  they  show  promise  in  that  they  attend  to  both  the 
logistic  and  the  human  aspects  of  maintenance  and  associated 
operations . 

Second,  they  show  a  limitation  in  their  handling  of  the 
human  aspect.  That  is,  they  dwell  entirely  on  maintenance 
time.  They  do  not  take  into  account  maintenance  errors . 

Finally,  they  appear  to  be  concerned  chiefly  with  improving 
the  cost-effectiveness  of  maintenance  at  a  given  level  of 
system  readiness.  Little  attention  is  paid  to  the  possi¬ 
bility  of  improving  system  readiness. 

In  contrast,  the  AMES  Model  focuses  primary  attention  on 
system  readiness  and  human-related  variables  contributing  to 
system  readiness.  The  model  is  useful  in  looking  for  ways 
to  improve  system  readiness  through  the  mechanism  of  a 
reduced  error  rate  among  maintenance  personnel. 
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SECTION  III 

METHOD  TO  IDENTIFY 
DESIGN  FEATURES  INFLUENCING 
MAINTENANCE  EFFECTIVENESS 


Task  III  of  the  DFM  project  had  two  purposes.  One  was  to 
identify  design  features  associated  with  maintenance  effec¬ 
tiveness.  The  other  was  to  obtain  an  estimate  of  the  gain 
that  could  be  achieved  if  the  design  problems  were  allevi¬ 
ated.  The  work  of  Task  III  is  described  under  the  following- 
major  headings: 

•  Basic  Planning 

•  Selection  of  Subsystems  for  Investigation 

•  Preparation  for  Investigation 

•  Performance  of  Investigation 


BASIC  PLANNING 

Basic  planning  covered  three  aspects  of  preparation  for  the 
work  of  Task  III.  The  first  concerned  the  method  of  esti¬ 
mating  achievable  gain.  The  second  concerned  the  selection 
of  subsystem  candidates  for  investigation.  The  third  con¬ 
cerned  the  method  of  treating  maintenance  effectiveness 
data. 


Plan  for  Estimating  Gain 


Assuming  that  maintenance  weaknesses  on  a  particular  sub¬ 
system  could  be  traced  to  features  of  system  design,  and 
assuming  corresponding  changes  in  design  were  implemented, 
some  improvement  could  be  expected  in  maintenance  effective¬ 
ness.  The  question  is,  how  much  improvement?  In  addressing 
that  question,  investigators  in  other  studies  have  made 
estimates  based  on  the  judgment  of  knowledgeable  maintenance 
personnel . 

After  some  consideration,  that  approach  was  rejected.  An 
alternate  source  of  information  was  found  within  the  F-14 
data  base  itself.  Represented  there  were  subsystems  showing 
strong  as  well  as  weak  maintenance  effectiveness.  It  was 
reasoned  that  records  of  strength  (in  terms  of  low  error 
rates  and  elapsed  maintenance  times)  were  just  as  valid  as 
records  of  weakness.  Records  of  strength,  therefore,  were 
designated  for  use  as  indications  of  the  levels  of  effec¬ 
tiveness  achievable  when  the  subsystems  are  properly 
designed. 
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Plan  for  Selecting  Subsystem  Candidates 

Given  the  decision  to  use  examples  of  both  strong  and  weak 
maintenance  effectiveness,  the  idea  of  pairing  was  adopted. 
A  plan  was  devised  to  select  a  group  of  strong  subsystems 
opposite  a  set  of  weak  subsystems,  with  each  "( strong/ weak) 
pair  matched  by  type  of  hardware  involved.  It  was  further 
decided  to  confine  the  sample  to  one  work  center  instead  of 
five  as  originally  contemplated.  The  one' chosen  was  Work 
Center  220,  Electrical/Instrumentation . 


Plan  for  Treating  Maintenance  Effectiveness  Data 

Two  kinds  of  data  were  expected  from  the  study:  data  perti¬ 
nent  to  the  design  features  under  investigation,  and  data 
pertinent  to  the  impact  of  those  design  features  on  mainte¬ 
nance  effectiveness. 

It  was  planned  from  the  beginning  to  handle  design  feature 
data  in  narrative  form.  This  plan  remained  intact.  How¬ 
ever,  the  plan  for  treating  maintenance  effectiveness  data 
had  to  be  changed  because  of  the  decision  to  use  strength 
and  weakness  records  from  the  F-14  data  base. 

The  following  agreement  was  made  regarding  the  treatment  of 
maintenance  effectiveness  data.  Maintenance  weakness  would 
be  represented  by  the  actual  values  of  all  Type  I  and  Type 
Il/d  error  rates,  and  elapsed  maintenance  times  found  in  the 
data  base.  Maintenance  strength  would  be  represented  by  the 
averages  of  such  values,  as  obtained  for  the  examples  selec¬ 
ted  for  investigation. 

The  purpose  of  this  arrangement  was  to  prepare  for  the  com¬ 
parisons  to  be  made  in  Task  IV.  Records  reflecting  the 
weakness  condition  would  provide  a  baseline  against  which 
various  estimates  of  improvement  could  be  measured.  Esti¬ 
mates  of  improvement  would  be  made  possible  by  the  averaged 
values  of  strength. 


SELECTION  OF  SUBSYSTEMS  FOR  INVESTIGATION 

Subsystems  were  selected  for  investigation  in  two  steps. 
First,  a  listing  of  candidates  was  assembled.  Then  sub¬ 
systems  were  selected  for  actual  use.  In  the  process,  quan¬ 
titative  values  of  maintenance  strength  were  determined  for 
use  in  Task  IV. 
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Listing  of  Candidates 

Within  the  F-14  data  base,  maintenance  summary  records  from 
Work  Center  220  were  isolated  for  analysis.  A  search  was 
made  for  subsystems  with  the  following  characteristics: 

•  Large  number  of  corrective  maintenance  actions 

•  High  Type’  I  error  rate 

•  Low  Type  I  error  rate 

•  High  Type  Il/d  error  rate 

•  Low  Type  Il/d  error  rate 

•  High  elapsed  maintenance  time 

•  Low  elapsed  maintenance  time 

All  subsystems  had  to  show  the  first  characteristic.  With 
regard  to  the  remaining  characteristics,  each  subsystem  had 
to  show  at  least  one.  In  addition,  the  listing  had  to  be 
large  enough  to  contain  two  examples  of  each  kind  of  mainte¬ 
nance  strength  and  each  kind  of  weakness. 

The  final  listing  of  candidates  is  presented  in  Table  16. 

It  contains  22  subsystems.  Subsystems  are  identified  at  the 
left  and  maintenance  effectiveness  at  the  right. 

Subsystem  Identification.  The  meanings  of  WUC  (Work  Unit 
Code)  and  NAME  are  obvious . 

Maintenance  Effectiveness.  The  rates  for  Type  I  and  Type 
Il/d  errors  are  shown  as  percentaaes  of  R/R  actions. 

Elapsed  maintenance  times  (EMT)  are  shown  as  percentages  of 
R/R  actions  exceeding  3.5  manhours. 

Selection  of  Subsystems  for  Actual  Use 

From  the  listing  given  in  Table  16,  12  subsystems  were  se¬ 
lected  for  investigation.  As  shown  in  Table  17,  they  fell 
into  three  groups: 

•  Subsystems  with  high  and  low  Type  I  error  rates 

•  Subsystems  with  high  and  low  Type  Il/d  error  rates 

•  Subsystems  with  high  and  low  elapsed  maintenance  times 

Each  group  contained  two  examples  of  its  respective 

conditions. 

Within  each  group,  provision  also  was  made  for  the  matching 
of  subsystems  (strong  vs.  weak)  having  similar  types  of 
hardware.  Thus,  the  Lateral  Control  System,  WUC  142 
(showing  strong  maintenance)  was  matched  with  the  Longi¬ 
tudinal  Control  System,  WUC  144. (showing  weak  maintenance); 
the  Approach  Power  Control  System  (strong)  was  matched  with 
the  Power  Plant  Installation  Associated  Eouipment  (weak) ; 
and  so  on . 
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TABLE  16.  CANDIDATES  FOR  INVESTIGATION 

SUBSYSTEM  Z  ERRORS  EMT 


wuc 

NAME 

I 

Il/d 

Z>3. 

138 

Wheel  Brake  System 

17.1 

9.0 

5 

139 

Nose  Wheel  Steering 

0.0 

4.2 

32 

142 

Lateral  Control  System 

0.0 

3.8 

45 

144 

Longitudinal  Control  System 

22.6 

7.4 

16 

146 

Flap/ Slat  Control  System 

0.0 

12.2 

42 

148 

Wing  Sweep  System 

0.0 

9.2 

34 

292 

Air  Inlet  Control  System 

12.8 

3.5 

32 

293 

Power  Plant  Control  System 

23.8 

4.9 

18 

29C 

Approach  Power  Control  System 

10.7 

4.0 

14 

29X 

Power  Plant  Installation  Assoc.  Eq. 

32.5 

4.8 

8 

421 

AC/DC  Power  System 

0.0 

9.7 

18 

425 

Misc.  Electrical  System  Comp. 

0.0 

7.4 

28 

49X 

Misc.  Utility  Associated  Equipment 

21.5 

6.1 

3 

511 

Flight  Instruments 

0.0 

9.9 

8 

512 

Navigation  Instruments 

6.8 

5.9 

6 

513 

Engine  Instruments 

30.6 

6.0 

13 

514 

Position  Indicators 

29.4 

6.0 

13 

515 

Utility  System  Indicators 

14.4 

6.4 

25 

564 

Attitude  Hdg .  Ref.  Set/Mag.  Det . 

0.0 

5.8 

13 

56X 

Flight  Reference  Assoc.  Equipment 

22.1 

4.5 

8 

573 

Wing/Flap/Glove  Vane  Control  Set 

12.2 

4.9 

18 

577 

Auto  Flight  Control 

25.6 

4.7 

11 

60 


203C.81.6 


NADC-79218-60 


TABLE  17.  SUBSYSTEMS  SELECTED  FOR  INVESTIGATION 


SUBSYSTEMS  WITH  HIGH  AND  LOW  TYPE  I  ERROR  RATES 


MAINT. 

EFFECT. 

wuc 

SUBSYSTEM  NAME 

TYPE  I 
ERROR  RATES 

Strong 

142 

Lateral  Control  System 

0.0 

Weak 

144 

Longitudinal  Control  System 

22.6 

Strong 

29C 

Approach  Power  Control  System 

10.7 

Weak 

29X 

Power  Plant  Installation  Assoc •  Eq. 

32.5 

SUBSYSTEMS  WITH  HIGH  AND  LOW  TYPE  Il/d  ERROR  RATES 


MAINT. 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

TYPE  Il/d 
ERROR  RATES 

Strong 

292 

Air  Inlet  Control  System 

3.5 

Weak 

146 

Flap/ Slat  Control  System 

12.2 

Strong 

512 

Navigation  Instruments 

5.9 

Weak 

511 

Flight  Instruments 

9.9 

SUBSYSTEMS  WITH  HIGH  AND  LOW  ELAPSED  MAINTENANCE  TIMES 


MAINT. 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

Z  ACTIONS 
>3.5  HRS. 

Strong 

138 

Wheel  Brake  System 

5.0 

Weak 

139 

Nose  Wheel  Steering 

32.0 

Strong 

56X 

Flight  Reference  Assoc.  Eq . 

8.0 

Weak 

573 

Wing/Flap/Glove  Vane  Control  Set 

18.0 
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The  purpose  of  the  matching  was  to  establish  a  basis  for 
comparison  later  in  the  investigation. 


Determination  of  Quantitative  Values  Denoting  Maintenance 
StrengtK 


Once  the  subsystem  selections  were  made,  it  became  possible 
to  determine  the  quantitative  values  needed  to  denote  main¬ 
tenance  strength  relative  to  Type  I  errors,  Type  Il/d  er- 


rors 

,  and 

elapsed  maintenance  time.  Those 

values 

• 

Type 

I  error  rate  (percentage) 

5.4 

• 

Type 

Il/d  error  rate  (percentage) 

4.7 

• 

EMT 

(percentage  of  actions  >3.5  hours) 

6.5 

Each  value  represents  a  simple  average  of  two  prior  values. 
The  basis  of  computation  is  shown  in  Table  18.  Note  that  in 
this  step,  pairs  of  subsystems  were  brought  together  in  a 
way  that  differed  from  the  matching  explained  earlier.  The 
reason  is  that  a  different  purpose  was  served  in  each  case. 

The  information  presented  in  Table  18  was  set  aside  for  use 
in  Task  IV,  where  the  effects  of  postulated  improvements  in 
maintenance  effectiveness  were  to  be  calculated  by  the  AMES 
model . 


PREPARATION  FOR  INVESTIGATION 

The  plan  called  for  technical  discussions  with  maintenance 
personnel,  aimed  at  discovering  which  design  features  had 
the  most  impact  on  maintenance  effectiveness.  Preparation 
for  those  discussions  occurred  in  two  stages.  One  was  the 
establishment  of  a  point  of  contact.  The  other  was  the 
development  of  an  interview  plan. 


Point  of  Contact 

The  closest  base  operating  and  maintaining  the  F-14  was  NAS, 
Miramar.  We  visited  Miramar  and  made  contact  with  Commander 
Gibb  Patterson,  COMFIT  Maintenance  officer.  Also  partici¬ 
pating  in  the  discussions  were  Commander  John  Mathews  and 
Lieutenant  Commanders  Ed  Pryor  and  Ray  Wegrin. 

These  people  were  familiar  with  Xyzyx ,  having  interacted 
with  us  on  past  contracts  involving  the  F-14  data  base  and 
the  portions  of  it  in  our  possession.  They  knew  of  the  AMES 
model.  We  explained  our  current  project  and  our  need  for 
their  help.  They  agreed  to  furnish  personnel  to  answer  our 
questions. 
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TABLE  18.  QUANTITATIVE  VALUES  DENOTING  MAINTENANCE  STRENGTH 


STRENGTH  IN  TYPE  I  ERRORS 


MAINT. 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

TYPE  I 

ERROR  RATES 

AVERAGE 

RATE 

Strong 

142 

Lateral  Control  System 

0.0 

Strong 

29C 

Approach  Power  Control 

10.7 

5.4 

STRENGTH  IN  TYPE  II /d 

ERRORS 

MAINT. 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

TYPE  II /d 
ERROR  RATES 

AVERAGE 

RATE 

Strong 

292 

Air  Inlet  Control  System^ 

3.5 

Strong 

512 

Navigation  Instruments 

5.9 

4.7 

STRENGTH  IN  ELAPSED 

MAINTENANCE  TIME 

MAINT. 

Z  ACTIONS 

AVERAGE 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

>3.5  HRS. 

RATE 

Strong 

138 

Wheel  Brake  System 

5.0 

Strong 

56X 

Flight  Reference  Assoc.  Eq.  8*0 

6.5 

63 


203C.81.6 


NADC-7 92 18-60 


We  provided  a  listing  of  the  subsystems  selected  for  inves¬ 
tigation.  Arrangements  were  made  for  a  series  of  visits  to 
Miramar  for  discussions  with  representatives  of  Work 
Center  220. 


Interview  Plan 

An  interview  plan  was  developed  to  assist  in  the  collection 
of  relevant  information  during  the  visits. 

The  interview  plan  is  summarized  in  Figure  2.  There  were 
two  objectives.  The  first  objective  was  to  explore  the 
influence  of  system  support  factors  on  the  maintenance  of 
all  subsystems  dealt  with  in  Work  Center  220.  This 
objective  was  served  by  Item  1.  The  second  objective  was  to 
identify  design  features  impacting  maintenance  effectiveness 
on  the  12  particular  subsystems  listed  in  Table  17.  This 
objective  was  served  by  Items  2  through  5  in  Figure  2. 

In  the  paragraphs  that  follow,  the  interview  plan  is 
described  under  the  following  headings: 

•  System  Support  Questionnaires 

•  Interview  Guides 

•  Design  Feature  Checklist 

•  Data  Collection  Sheet 

•  Interview  Instructions 


System  Support  Questionnaires 

System  support  was  dealt  with  by  a  pair  of  questionnaires. 
One  concerned  troubleshooting.  The  other  concerned  elapsed 
maintenance  time.  The  questionnaires  are  represented  here 
as  Exhibits  A  and  Br  respectively.  Since  the  two  are  com¬ 
pletely  alike,  except  for  the  subsystem  examples  they  cite, 
both  can  be  covered  effectively  by  a  single  explanation. 

Each  questionnaire  referred  to  F-14  maintenance  data  from 
Table  16  and  made  two  initial  points.  First,  the  data 
reflect  the  efforts  of  one  work  center  during  a  one— year 
period.  Variations  in  maintenance  resources  are  thus  highly 
constrained.  Second,  the  data  show  large  differences  be¬ 
tween  subsystems,  with  regard  to  elapsed  maintenance  time 
and  the  various  error  rates.  The  respondent  was  then  asked 
to  account  for  those  differences. 
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1.  EXPLORE  INFLUENCE  OF  SYSTEM  SUPPORT 

FACTORS 

•  Supervisory  leadership  • 

Troubleshooting 

•  Technician  capability 

• 

Corrective  maintenance 

•  Tech  pub  quality/ usability 

2.  IDENTIFY  DESIGN  FEATURES 

IMPACTING  TROUBLESHOOTING 

High  Type  I  Error  Rate 

• 

Lateral  Control  System 

• 

Approach  Power  Control  System 

Low  Type  I  Error  Rate 

• 

Longitudinal  Control  System 

• 

Power  Plant  Installation 

Assoc.  Eq. 

3.  IDENTIFY  DESIGN  FEATURES 

IMPACTING  ELAPSED  MAINTENANCE  TIME 

Many  Actions  >3.5  hrs. 

• 

Nose  Wheel  Steering 

• 

Wing/ Flap/Glove  Vane 

Control  Set 

Few  Actions  >3.5  hrs. 

• 

Wheel  Brake  System 

• 

•Flight  Reference  Assoc.  Eq. 

4.  IDENTIFY  DESIGN  FEATURES 

IMPACTING  CORRECTIVE  MAINTENANCE 

High  Type  d  Error  Rate 

• 

Flap/ Slat  Control  System 

• 

Flight  Instruments 

Low  Type  d  Error  Rate 

• 

Air  Inlet  Control  System 

• 

Navigation  Instruments 

5.  IDENTIFY  DESIGN  FEATURES 

IMPACTING  CHECKOUT 

High  Type  II  Error  Rate 

• 

Flap/ Slat  Control  System 

• 

Flight  Instruments 

Low  Type  II  Error  Rate 

• 

Air  Inlet  Control  System 

• 

Navigation  instruments 

Figure  2.  Summary  of  Interview  Plan 
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EXHIBIT  A 

QUESTIONNAIRE  ON  SYSTEM  SUPPORT  FACTORS 
FOR  TYPE  I  ERROR  RATE  CASES 


Table  1  lists  some  of  the  subsystems  maintained  in  this  work  center . 
The  column  marked  with  an  arrow  shows  the  error  rates  scored  by  tech¬ 
nicians  in  troubleshooting  the  various  subsystems  over  a  one-year 
period.  Note  that  in  some  cases,  the  error  rate  is  very  low,  while  in 
others  it  is  quite  high.  How  do  you  account  for  that?  Please  read 
on. 


TABLE  I.  CANDIDATES  FOR  INVESTIGATION 
SUBSYSTEM  %  ERRORS  EMT 


wuc 

NAME 

I 

Il/d 

%>3. 

138 

Wheel  Brake  System 

17.1 

9.0 

5 

139 

Nose  Wheel  Steering 

0.0 

4.2 

32 

142 

Lateral  Control  System 

0.0 

3.8 

45 

144 

Longitudinal  Control  System 

22.6 

7.4 

16 

146 

Flap/ Slat  Control  System 

0.0 

12.2 

42 

148 

Wing  Sweep  System 

0.0 

9.2 

34 

292 

Air  Inlet  Control  System 

12.8 

3.5 

32 

293 

Power  Plant  Control  System 

23.8 

4.9 

18 

29C 

Approach  Power  Control  System 

10.7 

4.0 

14 

29X 

Power  Plant  Installation  Assoc.  Eq . 

32.5 

4.8 

8 

421 

AC/DC  Power  System 

0.0 

9.7 

18 

425 

Misc.  Electrical  System  Comp. 

0.0 

7.4 

28 

49X 

Misc.  Utility  Associated  Equipment 

21.5 

6.1 

3 

511 

Flight  Instruments 

0.0 

9.9 

8 

512 

Navigation  Instruments 

6.8 

5.9 

6 

513 

Engine  Instruments 

30.6 

6.0 

13 

514 

Position  Indicators 

29.4 

6.0 

13 

515 

Utility  System  Indicators 

14.4 

6.4 

25 

564 

Attitude  Hdg .  Ref.  Set/Mag.  Det. 

0.0 

5.8 

13 

56X 

Flight  Reference  Assoc.  Equipment 

22.1 

4.5 

8 

573 

Wing/Flap/Glove  Vane  Control  Set 

12.2 

4.9 

18 

577 

Auto  Flight  Control 

25.  6 

4.7 

11 

1.  The  subsystems  themselves  may  vary  in  complexity  and  design,  thus 

making  some  easier  to  troubleshoot  than  others.  Could  that  be  the 
reason? _ 

2.  Supervisors  are  known  to  have  a  strong  impact  on  the  effectiveness 
of  their  people.  Supervisors  handling  the  Wing  Sweep  System,  for 
example,  might  have  shown  better  leadership  than  the  supervisors 
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EXHIBIT  A 
(continued) 

handling  The  Power  Plant  Control  System.  Could  that  have  hap¬ 
pened? _ Do  the  supervisors  in  this  work  center  special¬ 
ize  in  that  way? _  If  so,  how  are  the  subsystems  divided 

among  them?  Please  show  by  grouping  of  WUCs . 


3.  Technicians  vary  in  capability.  Those  assigned  to  Wing  Sweep  Sys¬ 
tem  might  have  been  brighter  or  better  trained  than  those  assigned 
to  the  Power  Plant  Control  System.  Could  that  have  happened? 
_  Are  the  technicians  within  this  work  center  special¬ 
ized  in  that  way? _  If  so,  how  are  the  subsystems  divid¬ 

ed  among  them?  Please  show  by  grouping  of  WUCs. 


4.  Tech  pubs  vary  in  quality  and  usability.  Those  written  for  the 
Wing  Sweep  System  might  have  been  better  than  the  ones  written  for 

the  Power  Plant  Control  System.  Could  that  have  happened? _ 

Do  the  manuals  differ  very  much? _ Are  they  used  enough 

for  the  differences  to  affect  quality  of  work? _ 

5.  In  summary,  please  complete  the  table  shown  below,  giving  your 
estimate  of  the  relative  contribution  of  the  factors  just  intro¬ 
duced.  State  your  estimate  in  the  form  of  a  percentage,  such  that 
all  entries  total  to  100  percent.  Note  that  space  is  provided  for 
some  other  possible  explanation.  If  you  know  of  one,  name  it  on 
Line  E. 


REASONS  WHY  SUBSYSTEMS  DIFFER 
IN  RECORDS  OF  MAINTENANCE  EFFECTIVENESS 
WITHIN  WORK  CENTER  220 


REASONS 


PERCENTAGE  OF  CONTRIBUTION 


A.  Design  for  maintainability 

B.  Supervisory  leadership 

C.  Technician  capability 

D.  Tech  pubs  quality  and  usability 

E.  Other? 
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EXHIBIT  B 

QUESTIONNAIRE  ON  SYSTEM  SUPPORT  FACTORS  FOR  EMT  CASES 

Table  1  lists  some  of  the  subsystems  maintained  in  this  work  center* 
The  column  marked  with  an  arrow  shows  the  percentage  of  high  elapsed 
maintenance  times  required  by  the  various  subsystems  over  a  one-year 
period*  Note  that  in  some  cases,  the  percentage  is  very  small,  while 
in  others  it  is  quite  large.  How  do  you  account  for  that?  Please 
read  on. 


TABLE  1.  CANDIDATES  FOR  INVESTIGATION 

SUBSYSTEM  Z  ERRORS  EMT 


wuc 

NAME 

i 

II /d 

%>3. 

138 

Wheel  Brake  System 

17.1 

9.0 

5 

139 

Nose  Wheel  Steering 

0.0 

4.2 

32 

142 

Lateral  Control  System 

0.0 

3.8  • 

45 

144 

Longitudinal  Control  System 

22.6 

7.4 

16 

146 

Flap/ Slat  Control  System 

0.0 

12.2 

42 

148 

Wing  Sweep  System 

0.0 

9.2 

34 

292 

Air  Inlet  Control  System 

12.8 

3.5 

32 

293 

Power  Plant  Control  System 

23.8 

4.9 

18 

29C 

Approach  Power  Control  System 

10.7 

4.0 

14 

29X 

Power  Plant  Installation  Assoc.  Eq . 

32.5 

4.8 

8 

421 

AC/DC  Power  System 

0.0 

9.7 

18 

425 

Misc.  Electrical  System  Comp. 

0.0 

7.4 

28 

49X 

Misc.  Utility  Associated  Equipment 

21.5 

6.1 

3 

511 

Flight  Instruments 

0.0 

9.9 

8 

512 

Navigation  Instruments 

6.8 

5.9 

6 

513 

Engine  Instruments 

30.6 

6.0' 

13 

514 

Position  Indicators 

29.4 

6.0 

13 

515 

Utility  System  Indicators 

14.4 

6.4 

25 

564 

Attitude  Hdg.  Ref.  Set/Mag.  Det. 

0.0 

5.8 

13 

56X 

Flight  Reference  Assoc.  Equipment 

22.1 

4.5 

8 

573 

Wing/Flap/Glove  Vane  Control  Set 

12.2 

4.9 

18 

577 

Auto  Flight  Control 

25.6 

4.7 

11 

1.  The  subsystems  themselves  may  vary  in  complexity  and  design,  thus 

making  some  easier  to  repair  than  others.  Could  that  be  the  rea¬ 
son? _ 

2.  Supervisors  are  known  to  have  a  strong  impact  on  the  effectiveness 
of  their  people.  Supervisors  handling  the  Wheel  Brake  System,  for 
example,  might  have  shown  better  leadership  than  the  supervisors 
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EXHIBIT  B 
(continued) 

handling  the  Lateral  Control  System.  Could  that  have  happened? 

_  Do  the  supervisors  in  this  work  center  specialize  in 

that  way? _  If  so,  how  are  the  subsystems  divided  among 

them?  Please  show  by  groupings  of  WUCs . 


3.  Technicians  vary  in  capability.  Those  assigned  to  Wheel  Brake 
System  might  have  been  brighter  or  better  trained  than  those  as¬ 
signed  to  the  Lateral  Control  System.  Could  that  have  happened? 
_ _ Are  the  technicians  within  this  work  center  special¬ 
ized  in  that  way? _  if  so,  how  are  the  subsystems  divided 

among  them?  Please  show  by  groupings  of  WUCs. 


4.  Tech  pubs  vary  in  quality  and  usability.  Those  written  for  the 
Wheel  Brake  System  might  have  been  better  than  the  ones  written 

for  the  Lateral  Control  System.  Could  that  have  happened? _ 

Do  the  manuals  differ  very  much? _ Are  they  used  enough 

for  the  differences  to  affect  quality  of  work? _ 


5.  In  summary,  please  complete  the  table  shown  below,  giving  your 
estimate  of  the  relative  contribution  of  the  factors  just 
introduced.  State  your  estimate  in  the  form  of  a  percentage,  such 
that  all  entries  total  to  100  percent.  Note  that  space  is 
provided  for  some  other  possible  explanation.  If  you  know  of  one, 
name  it  on  Line  E. 


REASONS  WHY  SUBSYSTEMS  DIFFER 
IN  RECORDS  OF  MAINTENANCE  EFFECTIVENESS 
WITHIN  WORK  CENTER  220 


REASONS  PERCENTAGE  OF  CONTRIBUTION 


A.  Design  for  maintainability 

B.  Supervisory  leadership 

C.  Technician  capability 

D.  Tech  pubs  quality  and  usability 

E.  Other? 
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Subsystem  design  was  offered  as  one  possible  explanation. 
Also  offered  were  three  factors  of  system  support:  super¬ 
visory  leadership,  technician  capability,  and  tech  pubs 
gual ity/usabil ity .  Each  was  explained  in  such  a  way  as  to 
get  the  respondent  to  think  before  reaching  a  conclusion. 

The  respondent  was  then  asked  to  estimate  the  relative  con¬ 
tribution  of  the  factors  introduced.  Space  was  provided  for 
an  additional  reason,  should  the  respondent  elect  to  name 
one . 

The  hypothesis  was  that  in  these  particular  situations,  most 
of  the  variability  in  maintenance  effectiveness  came  not 
from  system  support,  but  from  system  design. 

It  will  be  noted  that  the  system  support  questionnaires  made 
no  reference  to  Type  d  errors  or  Type  II  errors.  These  as¬ 
pects  of  maintenance  effectiveness  were  omitted  intention¬ 
ally.  The  assumption  was  that  the  effects  of  variations  in 
system  support  could  be  explored  adequately  through  Type  I 
errors  and  elapsed  maintenance  time  alone. 


Interview  Guides 

In  order  to  provide  structure  for  the  interviews,  two  guides 
were  developed.  One  concerned  problem  situations  (subsys¬ 
tems  with  high  error  rates  or  EMT) .  The  other  concerned 
non-problem  situations  (subsystems  with  low  error  rates  or 
EMT) .  The  interview  guides  are  represented  here  as  Exhibits 
C  and  D,  respectively.  Since  the  two  are  equivalent  except 
for  problem/non-problem  orientation,  they  will  be  covered 
with  a  single  explanation. 

Each  application  of  a  guide  focused  on  one  specific  subsys¬ 
tem  drawn  from  Table  17.  The  investigator  prepared  for  the 
interview  by  bringing  together  a  guide  sheet  and  a  Data 
Collection  Sheet  (Exhibit  F) .  On  the  data  collection  sheet, 
he  identified  the  subsystem  selected.  In  the  lead  statement 
of  the  guide,  he  inserted  a  relevant  phrase  to  particularize 
the  situation.  Relevant  phrases  came  from  the  following 
set : 


•  technicians  made  many  errors  in  troubleshooting 

•  technicians  made  few  errors  in  troubleshooting 

•  many  maintenance  actions  took  longer  than  3.5  hours 

•  few  maintenance  actions  took  longer  than  3.5  hours 

•  Technicians  made  many  errors  in  corrective  maintenance 

•  technicians  made  few  errors  in  corrective  maintenance 

•  technicians  made  many  errors  in  checkout  following 
repair 

•  technicians  made  few  errors  in  checkout  following 
repair 
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EXHIBIT  C 

INTERVIEW  GUIDE  FOR  PROBLEM  SITUATIONS 


The  record  shows  that  this  subsystem  was  hard  for  the  techicians  to 

work  with.  That  is,  _ . 

1.  What  is  there  about  system  design  that  could  have  made  the  work 
hard? 

2.  You  have  mentioned  _  as  an  aspect  of  design 

that  tends  to  make  the  work  hard  to  do.  Please  explain  further. 

3.  In  the  past  two  years,  has  this  subsystem  been  changed  in  any 

way? _ Has  the  change  made  the  work  harder  or  easier? 

4.  Please  explain  your  answer  to  Item  3. 

5.  If  you  had  the  power  to  make  other  changes  in  system  design,  what 
would  they  be? 

6.  What  other  aircraft  had  (or  has)  a  better  design  for  the  main- 
tainer,  on  the  functions  covered  by  this  subsystem? 

7.  Please  explain  your  answer  to  Item  6. 
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EXHIBIT  D 

INTERVIEW  GUIDE  FOR  NON-PROBLEM  SITUATIONS 


The  record  shows  that  this  subsystem  was  easy  for  the  technicians  to 

work  with.  That  is,  _ 

1.  What  is  there  about  system  design  that  could  have  made  the  work 
easy? 

2.  You  have  mentioned  _  as  an  aspect  of  de¬ 

sign  that  tends  to  simplify  the  work.  Please  explain  further. 

3.  In  the  past  two  years,  has  this  subsystem  been  changed  in  any 

way? _ Has  the  change  made  the  work  harder  or  easier? 


4.  Please  explain  your  answer  to  Item  3. 

5.  If  you  had  the  power  to  make  other  changes  in  system  design,  what 
would  they  be? 

6.  What  other  aircraft  had  (or  has)  a  better  design  for  the  maintain’ 
er,  on  the  functions  covered  by  this  subsystem? 

7.  Please  explain  your  answer  to  Item  6. 
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The  purpose  of  the  lead  statement  was  to  set  the  stage  for 
inquiry  into  the  design  features  of  the  object  subsystem. 
That  inquiry  was  structured  by  Questions  1  through  7 ,  which 
were  meant  to  be  read  to  the  respondent  by  the  investigator. 

Questions  1  and  2  sought  to  extract  detail  comments  regard¬ 
ing  subsystem  parts  and  corresponding  design  features  that 
influenced  maintenance  effectiveness. 

Questions  3  through  7  sought  further  comments,  from  view¬ 
points  slightly  different  from  the  first.  One  such  view¬ 
point  was  that  of  design  change.  The  other  was  that  of 
comparison  with  other  designs. 

Comments  from  the  respondent  were  intended  to  be  captured  on 
data  collection  sheets. 


Design  Feature  Checklist 

A  listing  was  prepared  showing  examples  of  design  features 
affecting  maintenance.  It  is  represented  here  as  Exhibit  E. 
The  listing  was  not  intended  as  a  full  account  of  all  possi¬ 
ble  design  features.  The  aim  was  merely  to  provide  assis¬ 
tance  to  the  investigator  conducting  the  interviews.  The 
listing  was  meant  to  help  him  formulate  questions  and 
process  the  comments  heard  in  return. 


Data  Collection  Sheets 

A  blank  form  was  provided  for  use  by  the  investigator  in  re¬ 
cording  the  comments  of  the  respondent.  It  is  represented 
here  as  the  Data  Collection  Sheet,  Exhibit  F.' 

Prior  to  the  interview,  the  investigator  prepared  a  data 
collection  sheet  for  each  subsystem  drawn  from  Table  17.  In 
addition  to  the  WUC  and  name  of  the  subsystem,  he  entered 
two  qualifiers:  situation  and  problem  orientation. 

Situation  was  denoted  by  one  of  the  following: 

•  Type  I  Error  Rate 

•  Type  II  Error  Rate 

•  Type  d  Error  Rate 

•  Elapsed  Maintenance  Time 
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EXHIBIT  E 

DESIGN  FEATURES  AFFECTING  MAINTENANCE 

1*  Parts  Not  Easily  Accessible  for  Checking 

2.  Checkout  Routine  Excessively  Long 

3.  Some  Modes  of  Operation  Hard  to  Check  on  Ground 

4.  LRU  BITE  Tests  Incomplete 

5.  Subsystem  Not  Fully  Covered  by  Malfunction  Indie. 

6.  Malfunction  Indie.  Fail  to  Distinguish  Between  LRUs 

7.  Some  Parts  of  Circuitry  Not  Covered  by  Test  Points 

8.  Circuitry  Hard  to  Trace 

9.  Test  Equipment  Inconvenient  or  Inaccessible 

10.  Circuitry  Subject  to  Intermittents 

11.  Parts  Not  Easily  Accessible  for  Corrective  Actions 

12.  Special  Tool  Required 

13.  Awkward  Work  Position  Required 

14.  Parts  Not  Well  Labeled 

15.  Delicate  Components  Given  too  Little  Protection. 

16.  Fasteners  Require  too  Much  Work 
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Problem  orientation  was  denoted  by  one  of  the  following: 

•  Problem:  High  error  rate  or  EMT 

•  Non-Problem:  Low  error  rate  or  EMT 

The  remainder  of  the  sheet  consisted  of  an  open  field  for 
use  in  conjunction  with  the  Interview  Guides,  Exhibits  C 
and  D. 


Interview  Instructions 

The  various  elements  of  the  interview  plan  were  tied 
together  by  a  set  of  written  procedures  and  a  listing  of 
materials.  The  result  is  represented  in  Exhibit  G  as  the 
Interview  Instructions. 

The  materials  listed  were  the  exhibits  described  earlier. 

The  procedures  were  sequential  statements  directing  the 
investigator  to  use  the  materials  in  certain  combinations  on 
subsystems  drawn  from  Table  17.  Table  17  was  made  part  of 
the  plan  for  the  convenience  of  the  investigator. 


PERFORMANCE  OF  INVESTIGATION 

The  investigation  is  described  under  the  headings  of  data 
collection,  data  reduction,  and  data  analysis. 


Data  Collection 


Data  collection  was  accomplished  by  three  members  of  the 
technical  staff  during  three  visits  to  NAS,  Miramar. 
Commander  Pryor  gave  them  access  to  a  group  of  highly 
qualified  technicians,  who  cooperated  fully  in  the 
investigation.  A  listing  of  the  technicians  appears  in 
Table  19. 

The  first  order  of  business  was  to  focus  attention  on 
hardware  design  as  opposed  to  the  various  support  factors. 
That  was  accomplished  through  the  use  of  the  questionnaires 
shown  in  Exhibits  A  and  B.  One  questionnaire  dealt  with 
problems  in  troubleshooting.  The  other  dealt  with  factors 
contributing  to  excessive  elapsed  maintenance  time.  All 
technicians  agreed  that  the  differences  in  maintenance  per¬ 
formance  on  the  subsystems  under  study  were  primarily  the 
result  of  design  differences  rather  than  differences  in 
personnel  or  supervisory  quality. 
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EXHIBIT  G 

INTERVIEW  INSTRUCTIONS 


MATERIALS 


Exhibit  A: 

Exhibit  B: 
Exhibit  C: 
Exhibit  D: 
Exhibit  E: 
Exhibit  F: 
Table  2:  ‘ 


Questionnaire  on  System  Support  Factors  for  Type  I  Error 
Rate  Cases 

Questionnaire  on  System  Support  Factors  for  EMT  Cases 

Interview  Guide  for  Problem  Situations 

Interview  Guide  for  Non-Problem  Situations 

Design  Features  Affecting  Maintenance 

Data  Collection  Sheet 

Subsystems  Selected  for  Investigation 


PROCEDURES 

1.  System  Support  Factors 

a.  Apply  questionnaire  from  Exhibit  A,  for  Type  I  error  rate 
cases.  Record  results  directly  on  questionnaire. 

b.  Apply  questionnaire  from  Exhibit  B,  for  EMT  cases.  Record  re¬ 
sults  directly  on  questionnaire. 

2.  Type  I  Errors 


a.  Apply  interview  guide  from  Exhibit  C,  for  two  problem  situa¬ 
tions.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 

b.  Apply  interview  guide  from  Exhibit  D,  for  two  non-problem  situ¬ 
ations.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 
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EXHIBIT  G 
(continued) 


EMC 

a.  Apply  interview  guide  from  Exhibit  C,  for  two  problem  situa¬ 
tions.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 


b.  Apply  interview  guide  from  Exhibit  D,  for  two  non-problem  situ¬ 
ations.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 


Type  d  Errors 

a.  Apply  interview  guide  from  Exhibit  C,  for  two  problem  situa¬ 
tions.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 

b.  Apply  interview  guide  from  Exhibit  D,  for  two  non-problem  situ¬ 
ations.  Use  design  features  from  Exhibit  E,  as  necessary. 
Record  results  on  Data  Collection  sheet,  Exhibit  F. 


Type  II  Errors 

a.  Apply  interview  guide  from  Exhibit  C,  for  two  problem  situa¬ 
tions.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 

b.  Apply  interview  guide  from  Exhibit  D,  for  two  non-problem  situ¬ 
ations.  Use  design  features  from  Exhibit  E,  as  necessary.  Re¬ 
cord  results  on  Data  Collection  Sheet,  Exhibit  F. 
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EXHIBIT  G 
(continued) 


TABLE  2.  SUBSYSTEMS  SELECTED  FOR  INVESTIGATION 


SUBSYSTEMS  WITH  HIGH  AND  LOW  TYPE  I  ERROR  RATES 


MAINT. 

EFFECT. 

wuc 

SUBSYSTEM  NAME 

TYPE  I 
ERROR  RATES 

Strong 

142 

Lateral  Control  System 

0.0 

Weak 

144 

Longitudinal  Control  System 

22.6 

Strong 

29c 

Approach  Power  Control  System 

10.7 

Weak 

29x 

Power  Plant  Installation  Assoc*  Eq . 

32.5 

MAINT. 

SUBSYSTEMS  WITH  HIGH  AND  LOW  TYPE  Il/d  ERROR  RATES 

TYPE  Il/d 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

ERROR  RATES 

Strong 

292 

Air  Inlet  Control  System 

3.5 

Weak 

146 

Flap/ Slat  Control  System 

12.2 

Strong 

512 

Navigation  Instruments 

5.9 

Weak 

511 

Flight  Instruments 

9.9 

SUBSYSTEMS  WITH  HIGH  AND  LOW  ELAPSED  MAINTENANCE  TIMES 

MAINT. 

1  ACTIONS 

EFFECT. 

WUC 

SUBSYSTEM  NAME 

>3.5  HRS. 

Strong 

138 

Wheel  Brake  System 

5.0 

Weak 

139 

Nose  Wheel  Steering 

32.0 

Strong 

56X 

Flight  Reference  Assoc.  Eq . 

8.0 

Weak 

573 

Wing/Flap/Glove  Vane  Control  Set 

.  18.0 
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TABLE  19.  WORK  CENTER  220  PERSONNEL  INTERVIEWED 


DATE 

TECHNICIAN 

RATING 

ORGANIZATION 

5-21-80 

Harding 

AECS 

VF-51 

Akridge 

AE1 

VF-124 

6-17-80 

Havlu 

AE2 

VF-124 

Castro 

AE1 

VF-124 

Babbcock 

AE1 

VF-1 

Morgan 

AE2 

VF-1 

6-18-80 

Meir 

AE1 

FRAMP 

Havlu 

AE2 

VF-124 
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Interviews  were  then  conducted  dwelling  on  the  12  problem 
and  non-problem  situations  identified  in  Table  17.  The 
interviewers  asked  questions  derived  from  the  Interview 
Guide  and  the  Design  Features  Checklist.  The  technicians 
responded  with  explanations  based  on  their  F-14  experience. 
In  some  cases,  the  aircraft  itself  was  used  to  clarify 
particular  points. 

The  interviewer  employed  the  data  collection  sheets  to  cap¬ 
ture  what  key  points  he  could.  However,  the  objective  was 
to  let  the  technician  talk  with  a  minimum  of  interruptions 
or  requests  to  slow  down.  Therefore,  a  parallel  means  was 
used  to  collect  technician  outputs.  That  means  was  a  tape 
recorder.  In  all,  570  minutes  of  discussion  were  recorded, 
on  nine  cassettes. 

All  data  were  brought  back  to  Canoga  Park  for  reduction  and 
analysis. 


Data  Reduction 


The  staff  members  played  back  their  respective  tapes  and 
transferred  relevant  bits  of  data  to  corresponding  data 
collection  sheets.  A  typical  data  collection  sheet  com¬ 
pleted  in  this  manner  is  shown  in  Figure  3. 

From  each  completed  data  collection  sheet,  a  matrix  was  de¬ 
veloped,  cross-referencing  subsystem  components  with  the 
listing  of  design  features  that  affect  maintenance.  An  ex¬ 
ample  of  a  completed  matrix  is  shown  in  Figure  4.  There, 
the  components  are  identified  by  WUC  and  name,  and  the  de¬ 
sign  features  are  referred  to  by  number. 

Each  cell  in  the  matrix  represents  an  interaction  between  a 
design  feature  and  a  component.  An  open  circle  (o)  means 
that  the  analyst  searched  his  records  for  some  mention  of 
that  interaction.  A  closed  circle  (•)  means  that  he  found 
one.  Each  closed  circle  is  described  further  in  the  numer¬ 
ically  encoded  statements  below. 

It  is  recognized  that  the  various  design  features  apply  dif¬ 
ferentially  to  the  maintenance  situations  under  analysis. 

For  example.  Item  16  (Fasteners  require  too  much  work)  could 
contribute  to  Type  d  errors  but  not  to  Type  I  errors,  while 
Item  8  (circuitry  hard  to  trace)  would  apply  in  the  opposite 
direction.  The  original  plan  was  to  use  this  fact  to  struc¬ 
ture  the  data  collection/reduction  processes. 
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DATA  COLLECTION  SHEET 

/dbMO  ftdf>  fS-fajt  Cerubof  fciicfricaJ.  drmj).  dfrsor  ?Ajd> /■€**. 

WUC  SUBSYSTEM  NAME  SITUATION  PROBLEM 

ORIENTATION 

tZatfLJf&Uj  A tbuitJ'/lukfeA-  Odls^)  U  -b  CU40SS  .  7 la,  (Wt-  ofaci. 

coal  Onti^ouM^  heSe.  4i  bt,  /tu^udea  d>  a££*ss  AcoitcJc  . 


IbldcAud  di  MUUudld  UH  OS<L  k/  . I&jLJS  Osvut  SuJatf.  MfSfet**-  U  J&Uy 

-i  cJal&Ic.  .  epuade.  l&y/slajs  W  ebseAi/e,  /&wi*£jl.  yl  M&ta/i'tx-* 

Ajsb^CMdica&s-i  da  •^tauiuly  ihc- pilot  do  Jt/jJL.  knAu)h ^aps^sk/s 

(Ut  duo uc.  Ci  lit.  d/uuxsidje*. .  CAuJc  (jOhdfftiS pdll&vc  auea.^ 

Mdl  shodi'll  ootlade.  ^uuldy  eoi^p&uud 

& 2*iSo7s  QdlSj  d  JaJt  etHKu/ffr’s  but  AtidtUeo  mlMoM u)tUd .  Mui/di 
aJtus  dec  cud  do  Aiuce/e  euad  Ap&cecd  do  A&ucsdzttd .  foc^usKS  cuadec. 
Oub  bj  jdfUjo&i  b)  a. ptobCutC.-  ddtfec  cMlcedS  Oc  idtAL 5  dMoi  Jt&oH  cMu-k^ 

-fadL^pliu4j  CJM/UcJf)S>  dnjd/<n  Unyjtoust/s.  sjjf  Okies  cue.  dto  Munf- 
6'jjtotis  Ccucpull Motes  ^pl/'ces  tuad pci?  Ciidc^ 6upx>lkd (s/x/bd)  tUneae/w. 

Jnne.  debtee.  jOcdlfaJas Mwt  do  be  dad’  duu  *«  d/u^dukcedtd^  d>  ki/p  ietlkdz- 

JeUtlbot -eut/tl- .  C S’cucs-  UishUltd Suikdes -J 

0  -' 

toteutA  *J  ’SuickA^s  eau  nance  C/ el*  £if&lxeu>  <?>  Ceued&t^  ia  stidyd acdjudd- 
■%t*d  A^ed)  n  wptspeelj  u' 

~TIul  tut  -yu>  wodj  dodlu}  AtjdC'W-  dkod wodedbeuwL  Mudec  ■  dkuuAu)  o*t diu- sdty~ 

/Ja  A  uhd  uA  d/flrt "h CtL  ^Ujrt-fzAVt.  /L  JtfJ .  A/irtA/£ASt/rx  UlA-df  //-sJ  /?  i*J  /**/**£•  /.  iMt  ,4  /»>< 


'  ■*v  '-»■  -  O  ^  '  ~  jf  - -  ,/r|  y  /*%/ mjL- 

t mM i  ivcuj  dvJL’UaJcc.ckd.  MM-  w/VLdjfj^Mild'-. 6 dates?  oxjl  ^"Mc. pecssutc  sweddej 


'Mtuuu/lj  because  dby  'Way  be  u4k/  o*.  MtUcpdaces  eyjdlua  <n  cdlec  dacaca^d' 


Figure  3.  Typical  Completed  Data  Collection  Sheet 


82 


203C.81.6 


NADC-79218-60 


i&OoOOOOOOOOOO 


•  0  0  0 


^ooooooo  ^o°o  0 

^OOOOOGO  000  00 


5;  %  O 


0*0 


1  ^  %  o  •  %  %  «  O  000  00 

<N  o  o  o  ooo  o  o©o  o  o 

■>.  — — — - - — — — . "  I  "  II  — 

CO&«000000  0000  oo 

•N. 

^N%0©000%*%%  %  * 

V - 

Hi  \J  I  *  0  0  0  0  %%*  %%  • 

V)«%oooo%%«%*  « 

^•OOOOOOOOOOO  0 

fOO©oOOOOOOOOO 
^OOOOOOOOOO  0  0 

\%0000«00  00  O  0 


£  p 
in  | 

I 


■S' 

5 

ja 

\  to 

*  I  % 

^  ^  ^  NS 

>  >  ^  4| 

'fc  ^  ,1 

x  ^  ^  Vi) 

S  ia  ^ 

$  45  v?  4 

5  £ 

^  V5  'O  'A 


nil! 

^  ^  -t  ^  ft 
>  v?  i  $ 

ti*  is  is 


_\  c*  *0  >  V\  NO  is  JO 

^  V>  Vo  Vo  Nft  V>  V)V\V)V^\o 

^  no  sn  vT)  vf\  nQ  vO  -sfr  nQ 


83 


203C.81.6 


Figure  4.  Example  of  Matrix  Used  in  Technical  Data  Reduction 


NADC-79218-60 


As  it  turned  out,  there  was  no  way  to  confine  the  techni¬ 
cians  to  the  target  situation.  They  talked  about  things 
they  considered  relevant.  Thus,  their  comments  often  ex¬ 
ceeded  the  planned  boundaries.  For  this  reason,  the  matrix 
shown  in  Figure  4  and  all  other  matrices  typically  contain 
cell  entries  and  explanations  that,  strictly  speaking, 
should  apply  elsewhere.  These  excursions  are  tolerable 
because  each  was  explained  by  the  analyst  in  terms  of  its 
"proper  situation." 

The  matrix  thus  was  used  as  a  data  organizer,  for  analysis 
to  be  done  later.  Aside  from  that  purpose,  the  principal 
reason  for  the  matrix  approach  was  to  force  the  analyst  to 
consider  every  possible  interaction  when  reviewing  his  work. 

Data  Analysis 


Data  analysis  proceeded  in  a  manner  designed  to  answer  two 
questions : 

•  Do  DFM  features  correlate  with  maintenance  performance? 

•  Is  the  interview  process  efficient? 

The  first  question  was  addressed  by  restructuring  the  data 
stored  in  the  matrices.  The  second  question  was  addressed 
by  comparing  the  matrix  data  to  historical  records  located 
in  the  F-14  maintenance  data  base. 


Design  Features 

To  check  on  the  correspondence  between  DFM  features  and 
maintenance  performance,  four  more  matrices  were  created, 
covering  Type  I  errors.  Type  II  errors,  Type  d  errors,  and 
elapsed  maintenance  time.  Each  matrix  was  constructed  as 
illustrated  in  Figure  5. 

At  the  left,  subsystems  were  ranked  by  the  quantitative  val¬ 
ue  of  the  key  variable  involved.  At  the  upper  right,  DFM 
features  were  always  listed  in  the  same  order.  This  list 
equates  to  the  design  features  cited  earlier  in  Exhibit  E 
and  in  the  first  set  of  matrices. 

Each  matrix  had  cells  in  which  to  place  significant  symbols. 
A  closed  circle  (•)  meant  the  feature  had  been  mentioned  by 
the  technician.  An  X  meant  the  feature  did  not  apply  to  the 
situation  under  analysis.  An  open  circle  (o)  meant  the  fea¬ 
ture  applied  but  had  not  been  mentioned  by  the  technician. 

When  a  matrix  was  complete,  the  pattern  of  closed  circles 
provided  a  visual  indication  of  correspondence  between  each 
DFM  feature  and  subsystem  maintenance  performance. 
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DESIGN  FEATURE  AFFECTING  MAINTENANCE 
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Figure  5,  Example  of  Matrix  Used  in  Analysis  of  Design  Features 
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Interview  Efficiency 

The  interview  process  is  generally  considered  to  be  ineffi¬ 
cient  unless  structured  in  some  significant  way.  In  the 
present  case,  an  opportunity  for  structure  is  provided  by 
the  F-14  data  base.  The  question  is,  would  such  structure 
produce  a  gain  in  the  information  yielded? 

To  answer  that  question,  the  F-14  data  base  was  queried  for 
components  involved  in  high  error  rate  and  EMT  situations. 
The  initial  matrices  were  then  checked  to  see  whether  or  not 
each  component  had  been  mentioned  by  the  technician. 

Wherever  a  component  had  not  been  mentioned,  the  omission 
was  taken  as  a  sign  of  weakness  in  the  interview  process 
that  could  have  been  prevented  by  better  structuring. 

In  this  analysis,  only  a  small  sampling  of  the  F-14  data 
base  was  employed.  Data  sources  were  as  follows: 

Type  I  Errors:  A799  (3M  Aviation  No  Defect  Item 

Analysis  Summary 

M50. 4790. A2 5551-01)  July  1977 

through  January  1978  (all  Navy) 

Type  Il/d  Errors:  3M  Data  Records  for  6  squadrons, 

June  1976  through  July  1977 

Elapsed  Maintenance  Time:  3M  Data  Records  for  6  squadrons, 

June  1976  through  July  i977 
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SECTION  IV 

METHOD  TO  EVALUATE  THE  IMPACT  OF  IMPROVEMENTS 

IN  MAINTENANCE 


The  purpose  of  Project  Task  IV  was  to  evaluate  the  impact  of 
potential  maintenance  improvements  on  operational  perform¬ 
ance.  This  was  accomplished  throuqh  the  use  of  a  dynamic 
simulation  model  called  AMES  (Aircraft  Maintenance  Effec¬ 
tiveness  Simulation) .  The  AMES  model  is  designed  to  exploit 
the  idea  that  maintenance  effectiveness,  as  measured  by 
error  rates  and  work  times,  has  an  impact  on  system-level 
performance,  as  measured  by  such  parameters  as  operational 
readiness,  mean  flight  hours  between  failures,  etc. 

Using  3-M  data,  an  historical  baseline  was  established,  re¬ 
flecting  fleet  records  of  maintenance  effectiveness  and  sys¬ 
tem  performance.  The  model  was  then  run  several  times  to 
reflect  various  changes  in  maintenance  effectiveness  that 
could  conceivably  result  from  improvements  in  design  for  the 
maintainer.  Results  were  then  examined  to  see  to  what  ex¬ 
tent  these  changes  in  maintenance  effectiveness  produced 
corresponding  changes  in  operational  performance. 


PRELIMINARY  DISCUSSION  OF  MEASUREMENT  PROBLEMS 

The  data  used  in  this  study  were  factual,  i.e.,  derived  from 
established  Navy  records.  In  and  of  themselves,  the  data 
present  no  drawbacks  in  either  constructing  or  accomplishing 
measurements.  However,  when  seeking  to  draw  inferences  from 
them  at  a  higher  level,  certain  problems  arise.  These 
problems  are  characterized  by  difficulty  in  showing  high- 
magnitude  differences  when  various  conditions  of  operation 
are  compared . 

In  this  particular  case,  the  difficulty  appears  to  be  rooted 
in  three  principal  causes.  One  is  that  existing  measures  of 
system  performance  contain  high  variability.  The  second  is 
the  conservative  error  rates  derived  from  existing  3-M  data. 
The  third  is  that  the  AMES  Model,  in  its  present  form,  is 
not  designed  for  optimum  sensitivity.  These  causes  are 
further  elaborated  in  the  paragraphs  that  follow. 


Variability  of  System-Level  Performance 

Many  factors  contribute  to  the  overall  performance  of  a 
squadron  or  fleet  of  aircraft.  Thus,  the  variance  of 
system-level  performance  measures  is  usually  very  high. 
Also,  no  single  statistic  adequately  describes  the  capa¬ 
bility  of  an  aircraft,  squadron,  or  fleet.  Unfortunately, 
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this  fact  is  usually  ignored  in  discussions  of  systems  by 
individuals  not  familiar  with  the  characteristics  of  operat¬ 
ing  weapon  systems .  Attention  is  usually  focused  on  opera¬ 
tional  readiness  with  little  or  no  attention  given  to  other 
statistics,  including  the  variance  of  operational  readiness. 
For  example,  using  a  month  as  a  basic  unit  of  time,  the 
average  and  standard  deviation  of  operational  readiness  was 
calculated  for  six  squadrons  of  F-14A  aircraft  over  a  year 
of  operation.  The  average  is  46.62  but  the  standard  devia¬ 
tion  is  10.84. 

Some  of  the  factors  contributing  to  the  operational  readi¬ 
ness  statistic  are: 

1.  Failure  rate  or  Mean  Time  Between  Failures  (MTBF) . 

2.  Spares  availability  which  in  turn  is  affected  by: 

a.  The  inventory  of  spares  and 

b.  Turn-around  time  of  spares  throuqh  intermediate 
maintenance  and/or  the  depot. 

3.  Number  and  quality  of  maintenance  personnel  available. 

4.  Mission  schedule.  This  item  is  especially  important. 

A  very  low-demand  mission  schedule  could  artificially 
inflate  the  operational  readiness  figure.  As  mission 
demands  increase,  more  flights  are  flown,  thereby 
creating  more  opportunities  for  failures.  Conse¬ 
quently,  a  high-demand  mission  schedule  will  tend  to 
reduce  operational  readiness  as  compared  to  a  low- 
demand  mission  schedule. 

5.  Available  facilities  and  tools. 

6.  Error  rates  and  Elapsed  Maintenance  Time  (EMT)  or  the 
more  commonly  known  Mean  Time  To  Repair  (MTTR) .  This 
factor  is  affected  by  Design-For- the-Maintainer  (DFM) , 
as  well  as  training  and  availability  of  effective  job 
performance  aids. 

The  factor  having  the  greatest  impact  on  operational  readi¬ 
ness  is  spares.  Currently,  spares  account  for  approximately 
half  of  the  F-14A's  42%  Not  Operationally  Ready  (NOR) 
status.  About  18%  of  all  corrective  maintenance  actions 
entail  a  delay  greater  than  five  days.  These  delays 
(greater  than  five  days)  average  38  days. 

An  item  often  not  given  proper  consideration  in  studies  of 
spares  is  the  high  level  of  interaction  between  spares  and 
errors  in  corrective  maintenance.  Maintenance  errors  in¬ 
crease  the  total  number  of  corrective  maintenance  actions  as 
well  as  increase  the  consumption  ot  spares’! 
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Type  I  error  increases  the  Corrective  Maintenance  Actions 
(CMA)  as  well  as  the  consumption  of  spares.  In  Type  I 
error,  a  good  unit  is  erroneously  removed  from  the  system. 
This  can  occur  in  two  ways.  In  one  case,  a  good  unit  is 
removed  but  the  malfunctioning  unit  is  still  in  the  system. 
In  this  case  the  work  has  to  be  repeated  at  some  later  time, 
and  at  least  one  additional  spare  has  been  consumed  unneces¬ 
sarily.  In  the  second  case,  the  malfunctioning  unit  is  re¬ 
moved  as  well  as  one  or  more  "good"  units.  The  work  does 
not  have  to  be  repeated,  but  at  least  one  additional  spare 
has  been  consumed . 

Type  II  error  is  the  failure  to  detect  a  malfunction.  This 
increases  CMA  but  does  not  increase  the  consumption  of 
spares.  Type  d  error  is  the  cause  of  damage  by  a  human  ac¬ 
tion.  This  action  obviously  increases  both  CMA  and  the  con¬ 
sumption  of  spares.  Thus,  all  errors  increase  CMA,  and 
Types  I  and  d  errors  increase  both  CMA  and  the  consumption 
of  spares. 


Limitations  of  3-M  Data 

Although  measures-  of  time-to-perform  and  error  can  and  have 
been  derived  from  3-M  data,  the  measures  thus  obtained  are 
often  highly  conservative.  For  example.  Type  I  errors 
(treating  a  good  unit  as  bad)  are  identified  only  by  the  3-M 
791  report,  i.e.,  "bench  check  OK."  This  report  covers  only 
Remove  and  Replace  (R/R)  actions. 

Analysis  of  maintenance  records  shows  that  Repair- in-Place 
(RIP)  actions  outnumber  R/R  by  almost  2  to  1.  Unfortunately 
Type  I  errors  during  RIP  actions  cannot  be  detected  with  3-M 
data . 

It  is  reasonable  to  assume  that  Type  I  errors  are  made  in 
RIP  as  well.  Even  if  Type  I  errors  in  RIP  are  assumed  to  be 
one-half  the  Type  I  errors  in  R/R,  considering  such  errors 
would  essentially  double  the  total  Type  I  errors. 

The  identification  of  Types  II  and  d  errors  has  also  been 
highly  conservative.  Thus,  the  actual  errors  are  antici¬ 
pated  to  be  considerably  higher  than  the  approximately  14% 
error  rate  noted  to  date  and  used  in  the  model  runs.  Pre¬ 
vious  studies  with  more  supportive  data  (for  the  Air  Force) 
resulted  in  an  error  rate  of  about  35%. 


Limitations  of  the  AMES  Model 


The  AMES  model  used  in  this  study  is  not  a  logistics  model, 
•ihus,  its  handling  of  spares  is  quite  limited .  This  limita¬ 
tion  is  not  of  much  concern  when  the  system  level 
performance  (e.g.,  operational  readiness)  is  very  high  and 
the  spares  contribution  to  downtime  is  relatively  low. 
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However,  the  interaction  with  spares  starts  to  become  an 
item  of  concern  when  the  contribution  of  spares  to  downtime 
is  as  high  as  the  current  22%.  As  a  result,  any  improvement 
potential  determined  with  the  model  will  tend  to  be  very 
conservative.  That  is,  the  model  will  underestimate  the 
effect  of  reducing  errors  since  it  will  not  account 
"properly"  for  the  benefits  accrued  from  reducing  spares 
consumption . 

Another  precaution  necessary  in  viewing  the  results  of  the 
model  is  its  "newness."  The  AMES  Model  was  recently  devel¬ 
oped  for  the  Navy  Training  Equipment  Center.  The  original 
model  development  plan  included  a  phase  to  systematically 
exercise  the  model  and  determine  its  variance  and  sen¬ 
sitivity.  For  various  reasons,  this  phase  was  never 
conducted. 

The  high  variance  of  system  performance  parameters  indicates 
the  tremendous  complexity  of  a  weapon  system ,  which  includes 
the  operational  aspects,  the  maintenance  aspects,  and  the 
logistics  aspects.  This  complexity  is  what  contributes  to 
the  high  variance.  This  complexity  is  also  the  reason  why  a 
model  is  needed  to  adequately  study  the  relationship  between 
subystem  variables  and  system  parameters.  However,  models 
are  also  quite  complex,  albeit  not  as  complex  as  the  system 
they  are  representing.  Consequently,  model  development  is 
usually  an  iterative  process  wherein  each  iteration 
increases  the  representativeness  of  the  model  as  well  as  its 
credibility.  Unfortunately,  the  AMES  model  has  not  had  time 
to  go  through  its  second  or  later  iterations.  This  was  the 
first  attempt  at  using  the  model. 


SUMMARY  OF  ANALYSIS  PROCEDURE 

The  analysis  procedure  used  consisted  of  three  basic  steps. 
First,  an  historical  baseline  was  established  reflecting  ac¬ 
tual  3-M  records.  Second,  a  sensitivity  analysis  was  con¬ 
ducted  by  running  the  model  with  various  adjustments  to  the 
input  data.  Third,  the  baseline  was  modified  to  take  into 
account  the  factors  discussed  earlier,  i.e.,  system  varia¬ 
bility  and  limitations  of  the  3-M  data  and  the  model. 


Historical  Baseline 


To  verify  that  the  model  agreed  with  actual  squadron  per¬ 
formance,  a  baseline  was  established  using  performance  time 
and  error  rates  derived  from  existing  3-M  data.  This  base¬ 
line  reflected  an  average  squadron,  based  on  a  sample  of 
six.  No  adjustments  were  made  to  the  input  data.  The  model 
simulated  24  months  of  squadron  operation.  The  same  mission 
profile  was  used  for  this  and  all  subsequent  runs. 
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Sensitivity  Analysis 

A  sensitivity  analysis  was  conducted  to  determine  the  ef¬ 
fects  of  adjusting  variables  that  would  be  impacted  by  im¬ 
proved  DFM.  These  variables  included  error  rates  (of  vari¬ 
ous  types)  and  Elapsed  Maintenance  Time  (EMT) .  The  model 
was  run  a  total  of  six  times ,  with  various  combinations  of 
adjustments  to  the  input  variables.  Each  of  the  six  runs 
encompassed  the  entire  aircraft,  and  was  not  limited  to  any 
individual  system.  In  summary,  the  adjustments  made  in  each 
of  the  six  runs  were  as  follows: 

Run  1:  All  errors  reduced  to  zero 

Run  2:  EMT  reduced  to  95%  2.5  hours,  all  errors 

reduced  95% 

Run  3:  All  errors  reduced  50% 

Run  4:  Type  I  errors  reduced  95% 

Run  5:  Type  Il/d  errors  reduced  95% 

Run  6:  EMT  reduced  to  95%  2.5  hours 

The  outputs  of  each  run  were  then  examined  to  determine  the 
effects  of  these  adjustments  on  various  system-level  per¬ 
formance  measures  such  as  Operational  Readiness  (OR) ,  Mean 
Flight  Hours  Between  Failure  (MFHBF) ,  and  Maintenance  Man¬ 
hours  per  Flight  Hour  (MMH/FH) . 


Modified  Baseline 


The  error  rates  derived  from  the  3-M  data  are  known  to  be 
conservative.  Therefore,  a  second  baseline  was  developed  to 
simulate  an  error  rate  75%  greater  than  that  of  the  histori¬ 
cal  baseline.  This  modified  baseline  was  then  compared  to 
the  historical  baseline  to  determine  whether  the  resulting 
change  in  system  level  performance  was  statistically 
significant . 
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SECTION  V 
RESULTS 


Results  of  this  project  are  presented  under  three  headings: 

•  Design  Features  Affecting  Maintenance 

•  Interview  Efficiency 

•  Results  of  Model  Runs 


DESIGN  FEATURES  AFFECTING  MAINTENANCE 

Design  features  affecting  maintenance  are  examined  in  Tables 
20,  22,  23,  and  25,  covering  Type  I  error.  Type  II  error. 
Type  d  error,  and  elapsed  maintenance  time  situations,  re¬ 
spectively.  Note  that,  throughout  the  discussion,  design 
features  are  expressed  negatively.  This  allows  them  to  be 
dealt  with  as  problem  conditions  wherever  they  appear. 


Type  I  Error  Situations 


Table  20  identifies  nine  design  features  that  theoretically 
should  affect  Type  I  errors  (symbol  •  or  o) ,  and  seven  fea- 
tures  that  theoretically  should  not  (symbol  X).  The  fea¬ 
tures  that  should  have  an  impact  are  listed  below. 

1.  Parts  Not  Easily  Accessible  for  Checking 

3.  Some  Modes  of  Operation  Hard  to  Check  on  Ground 

4.  LRU  BITE  Tests  Incomplete 

5.  Subsystem  Not  Fully  Covered  by  Malfunction 
Indicators 

6.  Malfunction  Indicators  Fail  to  Distinguish 
Between  LRUs 

7.  Some  Parts  of  Circuitry  Not  Covered  by  Test  Points 

8.  Circuitry  Hard  to  Trace 

9.  Test  Equipment  Inconvenient  or  Inaccessible 

10.  Circuitry  Subject  to  Intermittents 

Given  the  potential  for  effect,  two  questions  arise  in  each 
case.  First,  did  the  technician  report  it  as  a  problem  in 
the  interview?  Second,  is  the  problem  reflected  in  the  Type 
I  error  rate? 

The  first  question  is  answered  by  the  presence  of  the 
symbol  (•) .  Each  filled  circle  indicates  that  the  tech¬ 
nician  mentioned  that  problem  in  connection  with  that 
particular  subsystem. 
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For  the  subsystem  examined  in  this  study,  the  technician 
reported  only  eight  design  features  affecting  Type  I  errors. 
They  were: 

1.  Parts  Not  Easily  Accessible  for  Checking 
3.  Some  Modes  of  Operation  Hard  to  Check  on  Ground 

5.  Subsystem  Not  Fully  Covered  by  Malfunction 
Indicators 

6.  Malfunction  Indicators  Fail  to  Distinguish 
Between  LRUs 

7.  Some  Parts  of  Circuitry  Not  Covered  by  Test  Points 

8.  Circuitry  Hard  to  Trace 

9.  Test  Equipment  Inconvenient  or  Inaccessible 

10.  Circuitry  Subject  to  Intermittents 


The  second  question  is  answered  by  the  location  of  the 
symbol  (•)  relative  to  Type  I  error  value.  Each  such  symbol 
aligned  with  a  subsystem  having  a  high  error  rate  indicates 
that  the  effect  was  actually  felt  by  the  hardware.  A  symbol 
(o)  aligned  with  a  subsystem  having  a  low  error  rate  indi¬ 
cates  that  the  effect  was  not  felt.  ~ 
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TABLE  20.  MAINTENANCE  PER  FORMAN' 
FEATURES  IN  TYPE  I  ERROf 


' 

SUBSYSTEMS  RANKED  BY 

TYPE  I  ERROR  RATES 

1.  Parts  Not  Easily 

Accessible  for  Checking 

2.  Checkout  Routine 

Excessively  Long 

3.  Some  Modes  of  Operation 

Hard  to  Check  on  Ground 

wuc 

SUBSYSTEM  NAME 

TYPE  1 

ERROR 

RATES 

29X 

Power  Plant  Installation  Assoc.  Eq . 

32.5 

D 

X 

O 

144 

Longitudinal  Control  System 

22.6 

D 

X 

• 

56X 

Flight  Reference  Assoc.  Eq. 

22.1 

o 

X 

• 

138 

Wheel  Brake  System 

17.1 

o 

X 

O 

292 

Air  Inlet  Control  System 

12.8 

o 

D 

O 

573 

Wing/Flap/Glove  Vane  Control  Set 

12.2 

o 

X 

o 

29C 

Approach  Power  Control  System 

10.7 

D 

D 

o 

512 

Navigation  Instruments 

6.8 

o 

D 

o 

139 

Nose  Wheel  Steering 

0.0 

o 

D 

o 

mm 

Lateral  Control  System 

0.0 

o 

a 

0 

i 

146 

Flap/Slat  Control  System 

0.0 

D 

a 

o 

511 


Flight  Instruments 


0.0 


X 


o 


CE  PERFORMANCE  VERSUS  DESIGN 
N  TYPE  I  ERROR  SITUATIONS 


DESIGN  FEATURE  AFFECTING  MAINTENANCE 


2.  Checkout  Routine 

Excessively  Long 

3.  Some  Modes  of  Operation 

Hard  to  Check  on  Ground 

4.  LRU  BITE  Tests 

Incomplete 

5.  Subsystem  Not  Fully  Covered 
by  Malfunction  Indicators 

7.  Some  Parts  of  Circuitry 

Not  Covered  by  Test  Points  . 

8.  Circuitry  Hard  to 

Trace 

9.  Test  Equipment  Inconvenient 
or  Inaccessible 

10.  Circuitry  Subject  to 
Intermittents 
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13.  Awkward  Work  Position 

Requi red 

14.  Parts  Not  Well 

Label ed 

15.  Delicate  Components*  Given 

Too  Little  Protection 

16.  Fasteners  Require  Too 

Much  Work 
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The  correspondence  between  design  feature  and  Type  I  error 
rate  is  summarized  in  the  tabulation  below. 


Design  Feature 


Subsystem  Occurrence 
Type  I  Error  Rate 
High  Low  Total 


1. 

Parts  Not  Easily 
Accessible  for  Checking 

3 

2 

5 

3. 

Some  Modes  of  Operation 
Hard  to  Check  on  Ground 

2 

0 

2 

5. 

Subsystem  Not  Fully 
covered  by  Malfunction 
Indicators 

0 

1 

1 

6. 

Malfunction  Indie.  Fail 
to  Distinguish  Between 
LRUs 

4 

2 

6 

7. 

Some  Parts  of  Circuitry 
Not  Covered  by  Test 

Points  • 

1 

2 

3 

8. 

Circuitry  Hard 
to  Trace 

1 

0 

1 

9. 

Test  Equipment  Inconven¬ 
ient  or  Inaccessible 

2 

1 

3 

10. 

Circuitry  Subject  to 
Intermittents 

(General 

problem; 

0 

no  distinction) 

/ 

TOTAL 

13 

8 

21 

This  summary  indicates  that  two-thirds  (13)  of  the  occur¬ 
rences  behaved  as  expected,  while  the  remaining  third  (8) 
pointed  in  the  opposite  direction.  That  is,  they  aligned 
themselves  with  subsystems  having  low  Type  I  error  rates. 
Five  design  features  fell  into  this  latter  category.  A 
brief  accounting  of  the  resulting  occurrences  is  given  in 
Table  21. 
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TABLE  21.  NEGATIVELY  EXPRESSED  DESIGN  FEATURES  ALIGNED  WITH 
SUBSYSTEMS  HAVING  LOW  TYPE  I  ERROR  RATES 


DESIGN  FEATURE 


SUBSYSTEM 


EXPLANATION 


1.  Part  Not  Easily 
Accessible  for 
Checking 

146  Flap/ Slat 
Control  System 

Auxiliary  Flat  Retract 
Switch  hidden  by  Cove 
Door  and  other  compo¬ 
nents.  However,  circuit 
status  readily  discern- 
able  by  visual  observa¬ 
tion  of  Auxiliary  Flap. 

511  Flight 
Instruments 

Pitot  Static  Probe  dif¬ 
ficult  to  access  for 
static  line  leak  check. 
When  leaks  occur,  all 
connected  components  are 
affected.  However,  OBC 
points  to  Air  Inlet  Con¬ 
trol  Programmer  rather 
than  Flight  Instruments. 

5.  Subsystem  Not 

Fully  Covered  by 

Malfunction 

Indicators 

139  Nose  Wheel 
Steering 

Nose  Wheel  misalignment 
is  not  denoted  by  mal¬ 
function  indicators. 
Technicians  must  recon¬ 
cile  differences  with 
Hydraulic  Damper. 
Troubleshooting  method 
holds  down  Type  I  error 
rate  but  tends  to 
lengthen  EMT. 

6*  Malfunction 

Indicators  Fail 
to  Distinguish 
Between  LRUs 

146  Flap/ Slat 
Control  System 

OBC  identifies  problem 
area  only.  Technician 
must  isolate  faulty  LRU. 
Troubleshooting  method 
appears  to  be  effective. 

139  Nose  Wheel 
Steering 

Same  as  5  above. 

NADC-79218-60 


DESIGN  FEATURE 


7.  Some  Parts  of 
Circuitry  Not 
Covered  by  Test 
Points 


9.  Test  Equipment 
Inconvenient 
or  Inaccessible 


TABLE  21. 
(continued) 


SUBSYSTEM  EXPLANATION 


142  Lateral  Technicians  cannot  check 

Control  System  current  draw  of  Trim 

while  under  load.  Such 
a  check  would  make 
troubleshooting  easier. 
They  are  able  to  troub¬ 
leshoot  accurately  with¬ 
out  it  but  tend  to 
lengthen  EMT. 


146  Flap/ Slat 
System 


511  Flight 
Instruments 


Certain  degree  switches 
are  installed  in  such  a 
way  as  to  make  testing 
impossible  without  cut¬ 
ting  the  wires.  Wire 
cutting  enables  high 
quality  troubleshooting 
but  tends  to  lengthen 
EMT  and  increases 
Type  d  errors. 

TTV-205  Tester  unpopular 
among  technicians.  How¬ 
ever,  it  is  not  needed 
for  all  tests.  It  Is 
used  only  when  an  in¬ 
flight  malfunction  can¬ 
not  be  duplicated  on  the 
ground.  Tests  that  are 
necessary  are  time  con¬ 
suming  but  not 
inaccurate. 
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Type  II  Error  Situations 

Table  22  identifies  four  design  features  that  theoretically 
should  affect  Type  II  errors  (symbol  •  or  o) ,  and  12  fea¬ 
tures  that  theoretically  should  not  (symbpl  X) .  The  fea¬ 
tures  that  should  have  an  impact  are  listed  below. 

2.  Checkout  Routine  Excessively  Long 

3.  Some  Modes  of  Operation  Hard  to  Check  on  Ground 

9.  Test  Equipment  Inconvenient  or  Inaccessible 

10.  Circuitry  Subject  to  Intermittents 

For  the  subsystems  examined  in  this  study,  the  technician 
reported  all  four.  Furthermore,  a  high  proportion  of  the 
problems  reported  appear  to  be  felt  by  the  hardware.  The 
correspondence  between  design  feature  and  Type  II  error  rate 
is  summarized  in  the  tabulation  below. 

Subsystem  Occurrence 

Design  Feature  Type  II  Error  Rate 

High  Low  Total 


2. 

Checkout  Routine 
Excessively  Long 

1  0 

1 

3. 

Some  Modes  of  Operation 
Hard  to  Check  on  Ground 

1  0 

1 

9. 

Test  Equipment  Inconven¬ 
ient  or  Inaccessible 

2  1 

3 

10. 

Circuitry  Subject  to 
Intermittents 

(General  problem; 
no  distinction.) 

0 

Total 

4  1 

5 

This  summary  indicates  that  four  of  the  five  occurrences 
behaved  as  expected,  while  one  pointed  in  the  opposite  direc¬ 
tion.  That  is,  it  aligned  itself  with  a  subsystem  havinq  a 
low  Type  II  error  rate. 

The  design  feature  involved  is  Item  9,  Test  Equipment  Incon¬ 
venient  or  Inaccessible.  The  explanation  is  similar  to  that 
already  given  relative  to  Type  I  errors.  That  is,  the  TTV-205 
Tester  is  not  needed  for  all  tests.  It  is  used  only  when  an 
in-flight  malfunction  cannot  be  duplicated  on  the  ground. 

Tests  that  are  necessary  are  time  consuming  but  not 
inaccurate. 
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TABLE  22.  MAINTENANCE  PERFORMANCE  7ER 
IN  TYPE  II  ERROR  SITUATIONS 


SUBSYSTEMS  RANKED  BY 
TYPE  II  ERROR  RATES 


WUC 


SUBSYSTEM  NAME 


TYPE  II 
ERROR 
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Longitudinal  Control  System 
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Type  d  Error  Situations 

Table  23  identifies  two  design  features  that  theoretically 
should  affect  Type  d  errors  (symbol  •  or  o)  ,  and  14  features 
that  theoretically  should  not  (symbol  X) .  The  features  that 
should  have  an  impact  are  listed  below. 

11.  Parts  Not  Easily  Accessible  for  Corrective  Actions 
15.  Delicate  Components  Given  Too  Little  Protection 

For  the  subsystem  examined  in  this  study,  the  technician  re¬ 
ported  both  features.  Only  a  modest  proportion  of  the  prob¬ 
lems  reported  appear  to  be  felt  by  the  hardware.  The  corre¬ 
spondence  between  design  feature  and  the  Type  d  error  rate 
is  summarized  in  the  tabulation  below. 

Subsystem  Occurrence 


Design  Feature 

Type  d  Error 
High  Low 

Rate 

Total 

11.  Parts  Not  Easily 

Accessable  for  Corrective 
Actions 

2 

2 

4 

15.  Delicate  Components  Given 
too  Little  Protection 

2 

2 

4 

Total 

4 

4 

8 

This  summary  indicates  that  only  half  (4)  the  occurrences 
behaved  as  expected,  while  the  other  half  (4)  pointed  in  the 
opposite  direction.  That  is,  they  aligned  themselves  with 
subsystems  having  low  Type  d  error  rates.  Two  design  fea¬ 
tures  fell  into  this  latter  category.  A  brief  accounting  of 
the  resulting  occurrences  is  given  in  Table  24. 


EMT  Situations 


Table  25  indicates  that  all  16  design  features  listed  theo¬ 
retically  could  affect  elapsed  maintenance  time.  The  tech¬ 
nicians  reported  all  except  two: 

4.  LRU  BITE  Tests  Incomplete 

13.  Awkward  Work  Position  Required 

Three  other  features  represent  general  problems,  with  no 
distinction  in  effect  from  system  to  system.  These  features 
are : 

10.  Circuitry  Subject  to  Intermittents 

14.  Parts  Not  Well  Labeled 

16.  Fasteners  Require  Too  Much  Work 
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That  leaves  11  features  with  differential  impact  on  EMT. 
These  features  and  their  correspondence  with  EMT  values  are 
summarized  in  the  tabulation  below. 

Subsystem  Occurence 

Design  Feature  %  High  EMT 

High  Low  Total 


1.  Parts  Not  Easily  4 

Accessable  for  Checking 

2.  Checkout  Routine  1 

Excessively  Long 

3.  Some  Modes  of  Operation  1 

Hard  to  Check  on  Ground 

5.  Subsystem  Not  Fully  4 

Covered  by  Malfunction 
Indicators 

6.  Malfunction  Indie.  Fail  2 

to  Distinguish  Between 

LRUs 

7.  Some  Parts  of  Circuitry  2 

Not  Covered  by  Test 

Points 

8.  Circuitry  Hard  to  Trace  1 

9.  Test  Equipment  Inconven-  2 

ient  or  Inaccessible 

11.  Parts  Not  Easily  2 

Accessible  for  Corrective 
Actions 

12.  Special  Tool  Required  1 

15.  Delicate  Components  Given  2 

too  Little  Protection 


2 

0 

0 

0 


1 


0 


0 

1 

0 


0 

2 


6 

1 

1 

4 


3 


2 


1 

3 

2 


1 

4 


22  7  29 


This  summary  indicates  that  three- fourths  (22)  of  the  occur¬ 
rences  behaved  as  expected,  while  one- fourth  (7)  pointed  in 
the  opposite  direction.  That  is,  they  aligned  themselves 
with  subsystems  having  a  low  EMT.  Five  design  features  fell 
into  this  latter  category.  A  brief  accounting  of  the 
resulting  occurrences  is  given  in  Table  26. 
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TABLE  23.  MAINTENANCE  PERFORMANCE  VERS 
IN  TYPE  d  ERROR  SITUAT'ONS 


* 

- 

SUBSYSTEMS  RANKED  BY 

TYPE  d  ERROR  RATES 

ts  Not  Easily 
sssible  for  Checking 

2.  Checkout  Routine 

Excessively  Long 

3.  Some  Modes  of  Operation 
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-  NCE  VERSUS  DESIGN  FEATURES 
"DNS 


DESIGN  FEATURE  AFFECTING  MAINTENANCE 


3.  Some  Modes  of  Operation 

Hard  to  Check  on  Ground 

4.  LRU  BITE  Tests 
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5.  Subsystem  Not  Fully  Covered 
by  Malfunction  Indicators 
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Trace 
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11.  Parts  Not  Easily  Accessible 
for  Corrective  Actions 

12.  Special  Tool 
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Required 
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15.  Delicate  Components  Given 
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TABLE  24.  NEGATIVELY  EXPRESSED  DESIGN  FEATURES  ALIGNED  WITH 
SUBSYSTEMS  HAVING  LOW  TYPE  d  ERROR  RATES 

DESIGN  FEATURE  SUBSYSTEM  EXPLANATION 


11.  Parts  Not  Easily  292  Air  Inlet 

Accessible  for  Control  System 

Corrective  Actions 


573  Wing/Flap/ 
Glove  Vane 
Control  Set 


15.  Delicate  Compo-  292  Air  Inlet 

nents  Given  Too  Control  System 

Little  Protection 


56x  Flight 
Reference 
Associated 
Equipment 


Delta  Pressure  Sensor 
on  starboard  side  is 
difficult  to  replace. 
However,  because  this 
component  rarely  fails, 
it  exerts  little  influence 
on  the  Type  d  error  rate. 

Mounting  screws  on 
Wing/Flap/Glove  Vane 
Control  hard  to  reach  with 
screwdriver .  However , 
this  problem  normally 
results  in  lengthened  EMT 
rather  than  Type  d  error. 

Nylon  static  line  fittings 
easily  damaged  by  over- 
torqueing.  When  detected 
immediately,  problem  may 
result  in  static  line 
leak,  which  can  be  mis¬ 
taken  for  a  malfunction 
elsewhere  in  system. 
Example:  Type  I  error  in 
Air  Inlet  Control 
Programmer . 

Same  as  above. 
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TABLE  25.  MAINTENANCE  PERFORMANCE  VERS 
IN  ELAPSED  maintenance  time  sn 


r*  - - - - - 

SUBSYSTEMS  RANKED  BY 

PERCENT  ELAPSED  MAINTENANCE  TIMES 
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TABLE  26.  NEGATIVELY  EXPRESSED  DESIGN  FEATURES  ALIGNED 

i 

WITH 

SUBSYSTEMS  HAVING  LOW  EMT 

DESIGN  FEATURE 

SUBSYSTEM 

EXPLANATION 

( 

1.  Parts  Not  Easily 

29x  Power  Plant 

Certain  connectors  on 

* 

Accessible  for 

Installation 

engine  firewall  are  out 

/ 

Checking 

of  reach  due  to  absence  of 
an  access  hole.  Use  of 

*. 

i 

i 

1 

r 

i 

substitute  test  points 
produces  an  increase  in 
troubleshooting  time. 

However,  the  increase  is 
not  great  enough  to  impact 
overall  EMT. 

r 

511  Flight 

Pitot  Static  Tube 

Instruments 

difficult  to  access 
for  static  line  leak 
check.  This  produces 

i 

r 

an  increase  in  trouble¬ 

! 

(  . 

shooting  time.  However, 
increase  not  great  enough 

r 

to  impact  overall  EMT. 

6.  Malfunction 

29x  Power  Plant 

OBC  identifies  problem 

Indicators 

Installation 

area  only.  Technicians 

r  - 

,j 

Fail  to 

must  isolate  faculty  LRU. 

l. 

Distingui sh 

Troubleshooting  consumes 

between  LRUs 

EMT  but  easy  replacement 
of  components  makes  up 
for  loss. 

9.  Test  Equipment 

511  Flight 

TTU-205  Tester  needed  for 

r " 

Inconvenient 

Instruments 

some  tests  but  not  all. 

i . 

or  Inaccessible 

Impact  therefore  felt 

4 

infrequently. 

11.  Parts  Not  Easily 

511  Flight 

Pitot  Static  Probe 

L  . 

Accessible  for 

Instruments 

difficult  to  replace. 

r 

i 

L 

Corrective  Actions 

However,  because  this 
component  rarely  fails, 
it  exerts  little  influence 
in  overall  EMT. 

f 

i 

15.  Delicate  Compo¬ 

56x  Flight 

Nylon  static  line  fittings 

L.  . 

nents  Given  Too  . 

Reference  Associ¬ 

easily  damaged  by  over- 

r 

Little  Protection 

ated  Equipment 

torqueing.  However,  since 
they  are  easily  replaced, 

L 

they  exert  little  in¬ 

r 

fluence  on  overall  EMT. 

i 

i 

L 
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INTERVIEW  EFFICIENCY 

Measurements  of  interview  efficiency  are  shown  in  Tables  27, 
28,  and  29,  covering  Type  I  error,  Type  Il/d  error,  and 
elapsed  maintenance  time  situations,  respectively. 


Type  I  Error  Situations 

Table  27  identifies  three  subsystems  discussed  in  the  inter¬ 
view  as  having  high  Type  I  error  rates.  Historical  records 
disclosed  that,  for  each  subsystem,  one  particular  component 
was  heavily  involved  in  the  "No  Defect"  reports. 

For  example,  the  Throttle. Control  Computer  had  been  replaced 
190  times  and  had  been  found  not  defective  30  of  those 
times.  The  Pitch  and  Mach  Trim  Actuator  had  been  found  non¬ 
defective  in  7  out  of  26  replacements.  The  Air  Inlet  Con¬ 
trol  Programmer  reflected  a  similar  condition  115  out  of  354 
times  it  was  replaced. 

All  three  components  denote  serious  maintenance  problems. 
Yet,  only  two  of  them  had  been  mentioned  by  the  technician 
during  the  interview.  This  suggests  that  the  interviewer 
did  not  focus  the  technician  well  enough  with  respect  to  the 
analysis  of  design  features  affecting  troubleshooting. 


Type  I I/d  Error  Situations 

Table  28  identifies  three  subsystems  discussed  in  the  inter¬ 
view  as  having  high  Type  II  and/or  Type  d  error  rates.  His¬ 
torical  records  disclosed  that  for  each  subsystem,  one  par¬ 
ticular  component,  or  group  of  components,  was  prominently 
involved  in  the  "Improper  Action"  reports. 

For  example,  the  Flap/Slat  Control  Electrical  Components  had 
been  cited  11  times,  the  Clock  three  times,  and  various 
Flight  Instrument  components,  from  one  to  three  times. 

Of  the  seven  items  named  in  the  historical  records,  only 
four  had  been  mentioned  by  the  technician  during  the  inter¬ 
view.  This  suggests  that  the  interviewer  did  not  focus  the 
technician  well  enough  with  respect  to  the  analysis  of  de¬ 
sign  features  affecting  checkout  and  other  maintenance 
actions . 


EMT  Situations 

Table  29  identifies  four  subsystems  discussed  in  the  inter¬ 
view  as  requiring  excessive  maintenance  time.  Historical 
records  disclosed  that  for  each  subsystem,  certain  compo¬ 
nents  required  disproportionate  amounts  of  time  to  maintain. 
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TABLE 

27.  INTERVIEW  EFFICIENCY  RELATIVE  TO 

TYPE  I  ERROR  SITUATIONS 

DOCUMENTED 

MENTIONED  IN 

wuc 

NAME 

NO 

DEFECTS 

INTERVIEW 

29C 

APPROACH  POWER  CONTROL  SYSTEM 

— 

— 

29C31 

Throttle  Control  Computer 

30  out  of  190 

NO 

144 

LONGITUDINAL  CONTROL  SYSTEM 

— 

— 

14441 

Pitch  and  Mach  Trim  Actuator 

7  out  of  26 

YES 

29X 

POWER  PLANT  INST.  ASSOC.  EQ. 

— 

— 

29X11 

Air  Inlet  Control  Programmer 

115 

out  of  354 

YES 

TABLE 

28.  INTERVIEW  EFFICIENCY  RELATIVE 

TO  TYPE  II /d  ERROR  SITUATIONS 

WUC 

NAME 

DOCUMENTED 

IMPROPER 

ACTIONS 

MENTIONED  IN 
INTERVIEW 

146 

FLAP/SLAT  CONTROL 

— 

— 

14765 

Flap/ Slat  Control  Elec.  Comp. 

ii 

YES 

512 

NAVIGATION  INSTRUMENTS 

— 

51211 

Clock 

3 

NO 

511 

FLIGHT  INSTRUMENTS 

— 

— 

51111 

Counter  Drum  Altimeter 

3 

YES 

51112 

Altimeter 

2 

NO 

51122 

Counting  Accelerometer 

1 

YES 

51132 

Mach  Speed  Indicator 

2 

NO 

51141 

Pitot  Static  Probes 

2 

YES 
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TABLE  29.  INTERVIEW  EFFICIENCY  RELATIVE  TO  EMT 

SITUATIONS 

wuc 

NAME 

AVERAGE  EMT 
(HRS/MANHRS) 

MENTIONED  ! 
INTERVIEW 

57  3E 

■  WING/FLAP/GLOVE  VANE 

CONTROL  SET 

573E1 

Wing/Flap/Glove  Vane  Control 

5.8/10.3 

YES 

573E2 

Electromechanical  Rotary 
Actuator 

3.3/13.0 

NO 

573E3 

Electromechanical  Rotary 
Actuator 

5.5/12.0 

NO 

139 

NOSE  WHEEL  STEERING 

— 

— 

13921 

Damper  ( Hydraul ic  s ) 

7.5/21.5 

YES 

142 

LATERAL  CONTROL  SYSTEM 

— 

— 

1424 

Lateral  Control  Electrical 
Components 

4.3/12.0 

NO 

14241 

Lateral  Transducer 

7.0/21.0 

NO 

14242 

Trim  Actuator 

6.0/12.0 

YES 

29C 

APPROACH  POWER  CONTROL 

— 

— 

29C33 

Linear  Elec*  Accelerometer 

12.0/36  ' 

NO 
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As  against  an  average  EMT  of  2.5  hours,  the  EMTs  shown  for 
the  components  in  this  sample  ranged  between  3.3  and  12 
hours.  Since  each  job  is  handled  by  more  than  one  tech¬ 
nician,  the  loading  in  manhours  is  even  higher.  That  value 
ranges  from  10.3  to  36. 

Of  the  eight  items  extracted  from  the  historical  records, 
only  three  had  been  mentioned  by  the  technician  during  the 
interview.  This  suggests  that  the  interviewer  did  not  focus 
the  technician  well  enough  with  respect  to  the  analysis  of 
design  features  affecting  elapsed  maintenance  time. 


RESULTS  OF  MODEL  RUNS 

One  hypothesis  of  this  study  was  that  adjusting  EMT  and 
error  rate  would  result  in  corresponding  changes  in  system- 
level  performance  measures.  Each  of  the  model  runs  de¬ 
scribed  in  Section  IV  was  therefore  examined  for  a  change  in 
the  following  parameters: 

•  Operational  Readiness 

•  Flight  Hours  per  Squadron 

•  Mean  Flight  Hours  Between  Failures 

•  Maintenance  Manhours  per  Flight  Hour 

The  changes  in  each  of  these  parameters  are  discussed  in  the 
paragraphs  that  follow. 


Changes  in  Operational  Readiness 

The  results  of  the  model  runs  with  regard  to  operational 
readiness  are  summarized  in  Table  30.  No  statistically 
significant  difference  was  noted  when  Elapsed  Maintenance 
Time  (EMT)  was  reduced  to  about  2.5  hours  or  less,  and 
errors  were  eliminated.  This  lack  of  statistical  sig¬ 
nificance  is  attributed  to  the  lack  of  adequate  sensitivity 
of  the  model  rather  than  to  any  lack  of  relationship  between 
technician  performance  and  operational  readiness. 

Part  of  the  problem  is  due  to  the  overly  conservative  meas¬ 
ures  of  error  rate.  As  discussed  earlier,  the  estimated  er¬ 
ror  rates  do  not  account  for  Type  I  error  committed  during 
almost  two- thirds  of  all  maintenance  actions.  Thus,  a  sec¬ 
ond  baseline  run  was  made  with  the  error  rate  increased  from 
the  14%  in  Baseline  1  to  about  25%.  This  run  is  designated 
as  Baseline  2. 

The  t-statistic  for  Baseline  2  is  for  the  comparison  between 
the  two  baseline  runs.  The  significance  indicates  that  a 
more  realistic  measurement  of  error  rate  is  needed.  Given  a 
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more  realistic  error  rate  in  the  baseline,  reduction  of  er¬ 
rors  would  probably  show  a  statistically  significant  im¬ 
provement  in  operational  readiness,  despite  the  problem  in 
model  sensitivity. 

In  order  to  improve  the  sensitivity  of  the  model,  the  same 
mission  demand  profile  was  used  for  each  week  simulated. 

This  approach  helped  to  reduce  the  standard  deviation  from 
about  10.84  in  the  actual  fleet  to  3.59  in  the  baseline  run 
—  a  reduction  of  about  67%. 

The  reduction  in  standard  deviation  demonstrates  one  of  the 
advantages  of  the  model.  That  is,  the  model  can  be  used  to 
study  the  isolated  effects  of  specific  variables.  Such  an 
approach  is  generally  not  possible  with  an  actual  system. 

Despite  the  reduced  variance,  the  results  in  Table  30  show 
that  the  model  has  not  been  sufficiently  fine-tuned  to  meas¬ 
ure  relatively  small  changes  in  operational  readiness. 

The  model  can  detect  larger  changes  in  operational  readi¬ 
ness.  A  set  of  runs  was  made  to  eliminate  any  delays  due  to 
spare  parts  or  facilities.  As  expected,  the  operational 
readiness  increased  considerably.  The  increase  was  from  an 
average  operational  readiness  of  53.7%  to  an  average  of 
81.1%.  This  is  an  increase  of  51%. 

As  mentioned  before,  maintenance  errors  increase  the  number 
of  Corrective  Maintenance  Actions  (CMAs)1  This  is  equiva¬ 
lent  to  decreasing  reliability  or  Mean  Time  Between  Failures 
(MTBF) .  It  appears  reasonable  to  assume  that  improving  re¬ 
liability  or  reducing  CMAs  should  improve  operational  readi- 
ness.  Thus,  we  concluded  that  the  lack  of  significance  is 
due  to  inadequate  sensitivity  of  the  model  to  small  changes 
in  operational  readiness.  Significant  changes  in  other  sys¬ 
tem  performance  statistics  are  discussed  later. 

An  alternative  method  was  used  to  estimate  the  relationship 
between  technician  performance  and  operational  readiness. 

The  method  selected  was  to  first  determine  the  effect  of  er¬ 
rors  on  CMAs,  and  then  determine  the  relationship  between 
CMAs  and  operational  readiness. 

The  curve*  in  Figure  6  was  used  to  estimate  the  extent  to 
which  operational  readiness  could  be  improved  by  improving 
system  reliability.  It  is  important  to  note  that  the  curve 
does  not  account  for  any  variance  in  either  Mean  Flight 
Hours  Between  Failures  (MFHBF)  or  operational  readiness. 


*R.L.  McGee  Reliability  and  maintainability  contribution 
to  Hornet  mission  success.  In  Proceedings 
1979  Annual  Reliability  and  Maintainability 
Symposium. 
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Figure  6.  Relationship  Between  Reliability  And  Operational  Readiness 
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Also,  the  curve  is  based  on  numerous  unstated  assumptions 
such  as  levels  of  spares  support,  mission  demands,  mean  time 
to  repair,  turn-around  time,  personnel  availability,  etc. 

As  these  factors  vary,  the  relationship  between  fleet  MFHBF 
and  operational  readiness  will  also  vary.  However,  the 
curve  can  be  used  to  establish  that  relationship,  under  the 
assumption  that  other  relationships  will  remain  relatively 
the  same  when  maintenance  errors  are  reduced. 

The  key  to  utilizing  the  curve  is  to  translate  both  the 
curve  and  performance  improvements  into  numbers  of 
corrective  maintenance  actions  (CMAs) .  Table  31  presents  a 
translation  of  the  curve  into  CMAs  and  MFHBF. 

The  first  two  columns  are  based  on  the  following: 

Mean  Flight  Hours  Between  Failures  (MFHBF)  equals  flight 
hours  divided  by  failures.  Failures  (as  measured  with 
field  data)  equals  total  CMAs  minus  cannibalization  CMAs. 
Also,  failures  consist  of  CMAs  due  to  actual  malfunctions 
and  CMAs  due  to  errors.  The  first  column  has  the  percent  of 
total  CMAs  due  to  errors.  The  second  column  shows  what  the 
MFHBF  would  be  if  the  respective  CMAs  due  to  errors  were 
reduced  to  0. 

The  .48  is  the  actual  MFHBF  measured  with  3-M  data.  The  .48 
includes  CMAs  due  to  errors.  The  reason  for  considering 
different  CMAs  is  the  previously  discussed  conservative  man¬ 
ner  used  in  identifying  maintenance  errors.  In  order  to  be 
consistent  with  our  conservative  approach  to  date,  the  first 
column  was  extended  only  to  include  a  possible  error  rate  of 
25%. 

The  third  column  shows  the  increase  in  MFHBF  if  the  cor¬ 
responding  possible  error  rate  in  Column  1  were  eliminated. 
The  final  column  shows  the  difference  of  operational  readi¬ 
ness  percent  from  the  start  point.  This  calculation  is 
based  on  an  interpretation  of  the  curve.  According  to  the 
curve,  each  hour  of  MFHBF  (on  the  lower  end  of  the  scale) 
represents  about  27%  of  operational  readiness.  Thus,  the 
increase  in  operational  readiness  is  determined  by  multiply¬ 
ing  27  by  the  delta  MFHBF,  e.g.,  27  times  .05  equals  1.4%. 

Obviously  the  model  is  not  sensitive  enough  to  detect  a 
change  of  operational  readiness  of  2.2%  to  4.0%.  It  is  im¬ 
portant  to  note  that  the  actual  system  has  an  even  larger 
variance,  and  therefore  will  require  an  even  larger  change 
to  achieve  statistical  significance. 
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It  is  also  important  to  note  that  a  small  change  of  a  weapon 
system  performance  parameter  represents  a  large  amount  in 
cost.  For  example,  consider  how  much  it  would  cost  to  in¬ 
crease  equipment  reliability  by  15%  to  25%.  It  appears  log¬ 
ical  to  assume  that  the  efforts  to  reduce  human  error  by  90% 
to  95%  would  cost  significantly  less. 


Changes  in  Flight  Hours  per  Squadron 

Table  32  shows  the  results  of  each  of  the  model  runs  in 
terms  of  flight  hours  per  squadron.  As  before,  two  baseline 
runs  are  provided.  Baseline  1  is  with  the  measured  error 
rates.  Due  to  the  highly  conservative  nature  of  the  error 
measurements,  a  second  run  was  made  with  an  error  rate  of 
approximately  25%. 

Run  1  shows  the  results  when  errors  are  eliminated.  The  im¬ 
provement  from  Baseline  1  is  13.7%.  Run  3  shows  the  results 
from  reducing  the  errors  by  approximately  one-half.  The  im¬ 
provement  from  Baseline  1  is  7.5%.  Using  Baseline  2  as  the 
reference,  the  improvements  for  90%  and  50%  would  be  28%  and 
21%,  respectively. 

Given  a  fleet  of  180  aircraft,  a  13.7%  increase  in  flight 
hours  of  existing  aircraft  is  equivalent  to  adding  24.66 
aircraft  (180  x  .137)  to  the  fleet.  At  a  cost  of  approxi¬ 
mately  $20  million  per  aircraft,  the  additional  aircraft 
represent  $493.2  million  or  nearly  half  a  billion  dollars. 
With  Baseline  2  as  the  reference,  the  figures  increase  to 
50.4  aircraft  and  $1,008  billion. 

If  the  improvements  include  error  reduction  as  well  as  lim¬ 
iting  elapsed  maintenance  time  to  no  more  than  2.5  hours  as 
simulated  by  Run  2,  the  improvement  in  flight  hours  in¬ 
creases  by  16%.  This  is  still  equivalent  to  adding  28.8 
aircraft  to  the  fleet  at  a  cost  of  approximately  $576  mil¬ 
lion  or  half  a  billion  dollars.  With  Baseline  2  as  the  ref¬ 
erence,  reducing  errors  only  (Run  3)  increases  the  flight 
hours  by  27.8%,  which  is  equivalent  to  50  aircraft  or 
$1  billion. 

Note  that  the  improvement  levels  for  all  runs  but  Run  4  are 
statistically  significant.  The  reason  for  Run  4  being  the 
exception  is  the  aforementioned  treatment  of  Type  I  errors. 
Type  I  errors  are  attributed  only  to  R/R,  which  account  for 
only  one- third  of  the  CMAs.  Thus,  reducing  the  Type  I  er¬ 
rors  to  near  zero  only  effects  one-third  of  the  CMAs.  In 
reality.  Type  I  errors  probably  occur  for  all  types  of  CMA 
resulting  from  some  diagnostic  action. 
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Changes  in  Mean  Flight  Hours  Between  Failure 

Table  33  shows  the  improvement  in  Mean  Flight  Hours  Between 
Failure  (MFHBF)  resulting  from  reducing  error  rates.  Reduc¬ 
ing  error  rates  decreases  CMAs  which  is  equivalent  to  in¬ 
creasing  system  reliability  or  MFHBF.  Using  Baseline  1  as 
the  reference,  eliminating  errors  is  equivalent  to  increas¬ 
ing  reliability  by  18.2%.  If  Baseline  2  is  used  as  the 
reference,  the  improvement  increases  to  37%. 


Changes  in  Maintenance  Manhours  per  Flight  Hour 

Table  34  shows  the  reduction  in  Maintenance  Manhours  per 
Flight  Hour  (MMH/FH).  In  view  of  the  current  shortage  of 
maintenance  technicians,  this  statistic  is  quite  important. 
By  eliminating  errors  only,  the  MMH/FH  can  be  improved  by 
13%.  By  reducing  both  errors  and  EMT  (Run  2),  the  MMH/FH 
improves  by  25.4%,  or  almost  double  the  improvement  possible 
by  eliminating  errors  alone.  This  is  about  the  same  amount 
of  reduction  from  Baseline  2  to  Run  1. 
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SECTION  VI 

CONCLUSIONS,  IMPLICATIONS, 
AND  RECOMMENDATIONS 


The  information  presented  in  this  report  leads  to  the 

following  conclusions,  implications,  and  recommendations. 

CONCLUSIONS 

1.  There  is  a  definite  correspondence  between  maintenance 
performance  and  DFM  problems  as  rationally  conceived. 
That  is,  negatively  expressed  DFM  features  align 
themselves  with  subsystems  having  high  error  rates  and 
high  EMT  far  more  often  than  with  subsystems  having  low 
error  rates  and  low  EMT. 

2.  DFM  problems  apply  differentially  to  the  various  error 
rate  situations.  That  is,  some  DFM  features  are  aligned 
with  Type  I  errors,  while  others  are  aligned  with  Type 
II  or  Type  d  errors. 

3.  All  DFM  problems  apply  to  EMT  situations.  These 
features  can  increase  EMT  even  when  the  error  rates 
themselves  are  not  changed. 

4.  A  DFM  feature  may  apply  in  a  given  situation  and  have  no 
significant  effect  on  error  rate  or  EMT.  The  reason  is 
that  other  factors  can  overcome  such  effects. 

5.  The  interview  process,  where  properly  structured,  shows 
a  great  deal  of  promise  as  a  way  of  gaining  insight  into 
DFM  problems.  The  principal  means  of  structure  is  prior 
knowledge  of  maintenance  history  and  system  operation, 
by  the  interviewers. 

6.  The  interview  process  could  also  be  improved  by  focusing 
on  engineering  personnel  as  well  as  maintenance  tech¬ 
nicians.  Maintenance  technicians  perform  effectively  as 
problem  sensors  but  not  as  subject  matter  experts. 

7.  The  methods  described  here  would  enable  analysts  to 
identify  and  describe  significant  DFM  problems  to  the 
advantage  of  system/ equipment  designers. 
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IMPLICATIONS 

The  results  discussed  above  have  some  important  implications 

for  the  DFM  program: 

1.  The  total  population  of  improvement  potentials  is 
huge,  i.e.,  equivalent  to  increasing  the  fleet  by 
7%  to  12%,  or  higher,  as  well  as  reducing  manpower 
demands  by  nearly  25%.  Thus,  a  smaller  number  of 
technicians  can  achieve  a  higher  level  of  perform¬ 
ance.  However,  this  potential  for  improving  the 
maintenance  technician's  performance  is  not  due  to 
DFM  alone. 

The  equipment  design  features  (from  DFM),  training, 
and  job  performance  aids  are  the  three  main  contribu¬ 
tors  to  improving  technician  performance.  At  the 
system  level  of  performance,  it  is  doubtful  that  the 
relative  contribution  of  each  can  be  isolated. 

Related  studies  indicate  that  it  is  possible  to  re¬ 
duce  errors  to  near  zero.  Under  study  conditions 
covering  only  one  or  two  of  the  three  items,  errors 
have  been  reduced  by  as  much  as  50%  to  97%. 

In  order  to  reduce  the  error  rate  to  near  zero  in  an 
operational  setting,  we  anticipate  that  all  three 
items  will  have  to  be  given  proper  attention.  No  one 
item  can  be  expected  to  be  so  all-encompassing  in  its 
effect  that  it  can  virtually  eliminate  errors. 

2.  Despite  the  huge  potentials,  the  actual  change  of 
system  statistics  is  relatively  small.  A  small 
change  in  system  parameters  represents  a  huge  amount 
in  dollars  because  of  the  tremendous  cost  of  modern 
weapon  systems.  Since  the  change  in  system 
statistics  is  relatively  small  and  system  variance  is 
usually  large,  field  data  will  not  necessarily 
reflect  an  actual  improvement  at  the  system  level  of 
measurement.  Eventually,  a  true  improvement  will  be 
reflected  in  the  statistics,  but  the  long  time 
required  for  measurement  may  not  be  practical. 

For  example,  improving  only  Work  Center  210 
(Electronics),  one  of  the  more  active  work  centers, 
will  reduce  the  number  of  improvements  to  about  25% 
of  those  noted  earlier.  This  reduces  the  flight  hour 
improvements  to  a  range  of  2%  (for  a  50%  reduction  of 
errors)  to  about  3%  (using  Baseline  1) .  Although 
this  still  translates  into  large  dollars,  field  data 
probably  will  not  reflect  the  improvements  for  a  long 
time . 
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3o  Relatively  small  changes  in  performance  time  or 
errors  will  have  no  appreciable  effect  on  system 
level  performance  parameters.  Thus,  attention  should 
be  focused  on  systems/ areas  exhibiting  high  error 
rates  which  can  be  attributed  to  related  causes, 
e .g . ,  inadequate  diagnostics  information.  Solving 
the  DFM  problem  should  be  reflected  in  the  mainte¬ 
nance  of  all  weapon  systems  sharing  the  same  problem 
and  sharing  the  solution. 

The  latter  point  is  important.  When  the  topics  are 
properly  selected ,  DFM  has  the  potential  of  affecting 
many  different  weapon  systems.  This  increases  the 
improvement  potential  to  all  the  weapon  systems  ame¬ 
nable  to  the  solution.  Thus,  the  potential  may  be 
savings  equivalent  to  $250  million  to  $1,008  million 
for  each  of  many  different  weapon  systems. 

4.  As  expected,  establishing  an  accurate  and  reliable 
measure  of  the  current  level  of  performance  is 
important.  Relying  on  3-M  data  alone  provides  an 
overly  conservative  measure  of  error  rate.  Corollary 
data  should  be  used  to  supplement  the  3-M  data.  It 
is  anticipated  that  an  accurate  measure  will  show  the 
error  rate  to  be  closer  to  30%  to  35%.  If  so,  DFM 
improvements  will  be  more  readily  seen  in  system  per¬ 
formance  measures  —  especially  with  the  model. 

5.  The  AMES  model  is  an  important  tool  in  helping  to  re¬ 
late  technician  performance  to  system  performance. 
However,  the  results  should  be  viewed  as  preliminary 
since  too  little  is  currently  known  about  the  inher¬ 
ent  characteristics  of  the  model.  The  model  needs  to 
be  exercised  more  to  determine  its  variance  and  sen¬ 
sitivity.  Also,  the  model  should  be  compared  to 
other  existing  models  to  determine  which  should  be 
utilized  in  future  efforts. 

6.  There  is  a  distinct  possibility  that  the  AMES  or  an 
equivalent  model  can  be  used  to  help  develop  a  mathe- 
rofticsl  expression  of  the  relationship  between  tech¬ 
nician  and  system  performances.  This  should  greatly 
facilitate  trade-off  studies  in  the  future.  More  im¬ 
portant,  it  should  help  put  human  factors  consider¬ 
ations  on  a  relatively  equal  footing  with  equipment 
considerations. 
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RECOMMENDATIONS 

This  project  resulted  in  the  development  of  a  feasible 
method  of  identifying  and  evaluating  improvements  in  design- 
for-the-maintainer.  The  applicability  of  this  method  needs 
to  be  studied  further.  It  is  recommended  that  such  a  study 
be  authorized  to  refine  the  investigative  method  along  the 
following  lines: 

1.  Extend  Level  of  Detail  of  Investigation.  The  project 
examined  sources  of  maintenance  error  rates  and  EMT 
at  the  subsystem  level.  The  3-M  data  should  be  pre¬ 
pared  and  probed  at  a  more  specific  level  of  detail/ 
i.e./  down  to  the  individual  component  level.  The 
data  are  capable  of  providing  this  level  of  detail. 
The  generation  of  information  on  design  factors 
affecting  maintenance  performance  at  such  a  level 
would  clearly  offer  systems  engineers  more  precise 
pointers  about  possible  design  improvements  than 
would  information  at  a  higher  level  of  generality. 

The  preparation  of  error  rate  data  at  this  level  of 
detail  would  also  improve  the  precision  with  which 
questions  could  subsequently  be  asked  —  and  answered 
—  during  the  interview  stage. 

2.  Modify  Preparation  of  Interviewers.  In  the  course  of 
this  project ,  it  was  found  that  tne  Xyzyx  staff  who 
interviewed  Navy  personnel  would  have  been  able  to 
conduct  this  task  more  effectively  if  they  had  been 
more  familiar  with  systems  they  were  investigating. 

One  of  the  major  problems  with  eliciting  information 
in  a  relatively  unstructured  interview  format  is  en¬ 
suring  the  relevance  of  the  information  generated; 
the  interviewer  is  better  able  to  do  this  if  he  can 
ask  probing  questions,  and  sift  the  answers  he  is 
given,  on  the  basis  of  some  degree  of  informed  system 
knowledge . 

It  is  suggested,  therefore,  that  future  investigative 
teams  undertake  a  short  informal  study  course  in  the 
operations  of  the  F-14  and/or  SH— 2F  aircraft _ systems 
to  be  studied.  This  course  of  study  should  include 
reading  the  manual,  meeting  with  engineers,  and 
attending  (if  possible)  Navy  training  sessions. 
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Revise  Investigative  Procedures  and  Techniques.  This 
project  indicated  that  a  number  of  improvements  could 
be  made  in  the  way  the  investigative  interviews  were 
handled.  Examples  of  the  kind  of  procedural  items 
needing  improvement  include: 

•  scheduling  of  interviews 

•  use  of  more  than  one  interviewer  in  each 
interview 

•  use  of  tape  recorders  to  assist  interviewers' 
recollection  of  points  made 

Interviews  need  to  be  scheduled  well  in  advance,  and 
after  careful  discussion  with  Navy  personnel  regard¬ 
ing  potential  interviewees  who  would  be  able  to  pro¬ 
vide  the  most  useful  information  about  the  relation¬ 
ship  between  maintenance  problems  and  system  design 
features.  In  the  detailed  scheduling,  it  is  impor¬ 
tant  to  ensure  that  interviews  are  scheduled  for 
times  when  these  individuals  can  be  guaranteed  (to 
the  extent  possible)  to  be  available. 

The  use  of  two  interviewers  is  greatly  preferable  to 
using  a  single  interviewer.  It  is  often  difficult 
for  one  person  both  to  pursue  an  unstructured  inter¬ 
view  dialogue  and  to  record  all  pertinent  data  pro¬ 
vided  by  the  respondent. 

Tape  recorders  can  offer  both  benefits  and  penalties 
m  an  interview  situation.  They  enable  a  complete 
record  of  the  interview  to  be  available  for  repeated 
analysis,  but  they  can  also  inhibit  the  spontaneity 
of  responses. 
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